Matomo github/docker repos

[halkeye]

Service(s)

infra.ci.jenkins.io, Docker Hub, GitHub

Summary

Original request I never finished - #2684 (comment)
I recreated the repo - https://github.com/jenkins-infra/docker-mamoto
Build is working - https://infra.ci.jenkins.io/job/docker-jobs/job/docker-mamoto/job/main/55/pipeline-console/?selected-node=113
Buuuuut, i guess container repo was also removed, so needs to be re-created

Also related to #3530

Reproduction steps

No response

[dduportal]

Why installing Matomo in our own cluster

• Manage data analytics ourselves instead of "giving" it to Google or any 3rd person
• Takeover control instead of relying on G.analytics (Ref. migrate google analytics to v4 #3530)

What

• Matomo (https://matomo.org/) is a PHP-based application to collect analytics data
• It's persistent storage is MariadDB / MySQL: https://matomo.org/faq/on-premise/matomo-requirements/
• Initial assesment for storage and scaling: migrate google analytics to v4 #3530 (comment)
• It needs an entrypoint (e.g. domain + HTTP + TLS). Proposal: analytics.jenkins.io domain which would be a CNAME to matomo.jenkins.io (which would point to the service itself)
  • 1 level of indirection to allow switching platforms in the future in a green blue deployment
• Baseline: hosting in the publick8s cluster with a managed MySQL database

How

• Need a Docker image to have an immutable "Matomo + set of plugins" artefacts - https://github.com/jenkins-infra/docker-mamoto
  • Avoid any need for a Persistent Volume
• Need an helm chart (either existing or a custom one)
• Need a managed MySQL database (in azure)
• Need DNS set up
• Need a deployment configuration release
  • cluster: publick8s
  • Setup and credentials to define (including database connection)
  • TLS enabled

[dduportal]

@halkeye do you see anything else?

[halkeye]

nothing else comes to mind

the first two how's are already done by me, https://github.com/halkeye-docker/matomo is way more up to date as I didn't have enough permissions to iterate on infra.ci

[halkeye]

Not sure the goals of this addition, but probably worth noting that another reason to switch is the built in gdpr support, if you use the existing database i have, you'll have most things anonymized, but especially referrals disabled (i noticed after a few days that peoples private jenkins installs were being logged to plugins.jenkins.io stats as referrals)

[dduportal]

Update:

• MySQL database created in production
• Docker image is now building and releasing and deployed to https://hub.docker.com/r/jenkinsciinfra/matomo

WiP:

• Support of ARM64 for the image
• Image updates tracking
• Helm chart

[dduportal]

Update:

• ARM64 image is available

[dduportal]

WiP on the helm chart: jenkins-infra/kubernetes-management#4032

[dduportal]

Update:

• Started to refresh Matomo - https://github.com/jenkins-infra/helpdesk/issues/3530 kubernetes-management#4032 by adding secrets in SOPS and setting up ARM64

WiP:

• Create a MySQL database in the public-db instance. Gotta use the https://registry.terraform.io/providers/petoju/mysql/latest/docs provider which is a fork of the original (and now archived) mysql providers https://registry.terraform.io/providers/winebarrel/mysql/latest and hashicorp's

[dduportal]

Update: database creation in jenkins-infra/azure#497

[dduportal]

Update:

• After a few misconfiguration, we succeeded in creating the database using Terraform:
  • hotfix 1: don't use the login value in the password field 😅 - jenkins-infra/azure@e970812
  • hotfix 2: force TLS for MySQL connection: jenkins-infra/azure@901d459
• Secrets updated with the database credentials.

WiP: installing an initial release without ingress

[dduportal]

Update: issue while starting the pod (cc @halkeye if it rings a bell or if you have some thoughts) with the following error:

+ '[' '!' -e matomo.php ']'
+ tar cf - --one-file-system -C /usr/src/matomo .
+ tar xf -
tar: .: Cannot utime: Operation not permitted
tar: .: Cannot change mode to rwxr-xr-t: Operation not permitted
tar: Exiting with failure status due to previous errors

• The bitnami Helm charts uses their own Docker image bitnami/matomo (as per https://github.com/bitnami/charts/blob/9792064632e5d6d795b4a530923188481127a2cc/bitnami/matomo/values.yaml#L62-L65)
• While our custom jenkinsciinfra/matomo image uses the official matomo base image as per https://github.com/jenkins-infra/docker-matomo/blob/bc8a08e42bf5b4240da0192a2815abd7722df6aa/Dockerfile#L1

=> There are quite some differences between both explaining our error

• The helm chart from bitnami enforces a non root user:

  • There is a security context defined for the main pod with runAsUser: 1001 and runAsNonRoot: true
  • Bitnami uses a non root user as per their documentation: https://docs.bitnami.com/tutorials/work-with-non-root-containers/

• The entrypoint of our image (and matomo official image) uses the WORKDIR /var/www/html which is own by the user www-data (UID 33)

  • It explains the error: the user 1001 is not allowed to write in the directory. This is the first problem

Easy to reproduce:

$ docker run --rm -u 1001 docker.io/jenkinsciinfra/matomo:0.1.1 ls -la
+ '[' '!' -e matomo.php ']'
+ tar cf - --one-file-system -C /usr/src/matomo .
+ tar xf -
tar: .: Cannot utime: Operation not permitted
tar: .: Cannot change mode to rwxr-xr-t: Operation not permitted
tar: Exiting with failure status due to previous errors

Gotta try ith the 33 user

[dduportal]

Specifying the user 33 goes further but fails due to the Apache configuration file being protected:

+ perl -pi -e 's/Listen\s+80$/Listen 8080/g; s{<VirtualHost *:80>}{<VirtualHost *:8080>}g' /etc/apache2/ports.conf /etc/apache2/sites-enabled/000-default.conf
Can't do inplace edit on /etc/apache2/ports.conf: Cannot make temp name: Permission denied.
Can't do inplace edit on /etc/apache2/sites-enabled/000-default.conf: Cannot make temp name: Permission denied

I believe we should try switching to the bitnami image

[halkeye]

I was trying to make the docker file diskless by including plugins in the docker file, especially good for renovate/updatecli type thing, but php apps have everything nested.

I'm pretty sure bitnami needs persistent disk and you do updates in the app, not the docker image.

I think most of the stuff in entrypoint can get moved to dockerfile instead of entryfile.

honestly the only part of entrypoint that needs to be on startup is envsubst to allow database env variables.

[dduportal]

I'm still not sure which way to go:

• bitnami helm chart have a lot of sane defaults for production (non root, config, etc.)
• but the official matomo image looks better.

Gotta try you suggestion because the bitnami image errors with

matomo 17:55:21.17 WARN  ==> The Apache configuration file '/opt/bitnami/apache/conf/httpd.conf' is not writable. Configurations based on environment variables will not be applied.
mkdir: cannot create directory '/opt/bitnami/apache/conf/bitnami/certs': Permission denied

[dduportal]

I'm still not sure which way to go:

* bitnami helm chart have a lot of sane defaults for production (non root, config, etc.)

* but the official matomo image looks better.

Gotta try you suggestion because the bitnami image errors with

matomo 17:55:21.17 WARN  ==> The Apache configuration file '/opt/bitnami/apache/conf/httpd.conf' is not writable. Configurations based on environment variables will not be applied.
mkdir: cannot create directory '/opt/bitnami/apache/conf/bitnami/certs': Permission denied

gotcha, it's running now (had to remove the 33 UID). Now: connectivity from the cluster to the MySQL DB. Alsmost there!

Thanks for the explanation @halkeye , it helps!

[dduportal]

OK, too much errors everywhere, I give up.

@halkeye do you have a recent installation working? I don't understand how the helm chart of bitnami can work with the setup in jenkins-infra/kubernetes-management#4032 : are you using this helm chart in your cluster?

It can't connect to the database while I can mysql and the custom entrypoint script of bitnami seems to have their own way of "health checking" but I'm unable to find a way to debug.

It seems there is a lot of thing: plugins in matomo, cronjobs that do whatever tasks, it's really hard to start this service without help.

[halkeye]

https://github.com/halkeye-docker/matomo is the version i eventually got running locally. I'm pretty sure i mentioned it in the task, but I eventually went to my own fork because it was too hard to test with infra.ci (I think it wasn't building or something, or i needed to actually tag it, can't remember now)

https://github.com/jenkins-infra/kubernetes-management/compare/helpdesk-3530-matomo-2?expand=1 is what I had locally when i was testing it. I don't know if its any different than what you have so far.

[dduportal]

https://github.com/halkeye-docker/matomo is the version i eventually got running locally. I'm pretty sure i mentioned it in the task, but I eventually went to my own fork because it was too hard to test with infra.ci (I think it wasn't building or something, or i needed to actually tag it, can't remember now)

https://github.com/jenkins-infra/kubernetes-management/compare/helpdesk-3530-matomo-2?expand=1 is what I had locally when i was testing it. I don't know if its any different than what you have so far.

Thanks Gavin! The problem is that this looks like a LOT of concepts (there are inline YAML for cronjobs doing things, plugins preinstalled in the helm chart, etc.). It's way more complicated to deploy and maintain in a (public) production context with so much moving pieces and no prior knowledge of the tooling.

Was there a particular reason to avoid the persistent volume? I would like to understand this as it might not be a constraint at all in the Jenkins case (I mean, azurefile are cheap and really good perfs. most of the time)

Besides, we are having weird issues with AKS, MySQL and ARM (on one side) and the bitnami helm-chart and MySQL on another.

What it means is that it will takes quite some time to bootstrap this in a "good enough for production" context and it needs some planning and back and forth to gain the required knowledge.

[halkeye]

Was there a particular reason to avoid the persistent volume? I would like to understand this as it might not be a constraint at all in the Jenkins case (I mean, azurefile are cheap and really good perfs. most of the time)

Honestly, it was done because originally at work we don't have access to storage in our internal deployments, but also because the bitnami image extracts the tarball into the persistent storage (kinda like jenkins plugin does for plugins). So to upgrade you need to click something in the UI that triggers an upgrade. Which means you can't use updatecli or renovate to trigger upgrades. Plus really all you need to persist is config.ini.php

Speaking of config.ini.php, you may want to use the one from my fork, I believe matomo is a bit picky about the config file, and if keys are missing, triggers a config page instead of the application itself.

There's also the matomo cloud option.

[lemeurherve]

Note: we should setup stats for every jenkins.io subdomain not analysed yet.