Symfony Messenger

[norkunas]

How to run Symfony Messenger with this setup?

I thought about this in compose.yaml:

  php-worker:
    build:
      context: .
      target: frankenphp_dev
    image: ${IMAGES_PREFIX:-}app-php
    restart: unless-stopped
    command: php bin/console messenger:consume async -v

Does it make sense? What are the options? Would be good to add something about this to README :)

[dunglas]

I use something similar in my projects.

This definitely is worth a documentation section!

[MrSpoocy]

Same here, I'm not sure if I create a second container or maybe frankenphp can handle the consumer in a second thread.

[norkunas]

noticed one thing with this approach, that all the images at startup tries to run database migrations

[MichaelBrauner]

I think that's totally fine.
Every container will try to run migrations.
As long as you have a single source of truth for your migrations (one database server for all containers) you should be fine.

[norkunas]

Yes, the first one to run wins :) but I am thinking for a flag that I'd set on the main container, and on messenger/scheduler containers not setting it then entrypoint could run based on that condition, is that possible?

[MichaelBrauner]

Yes, that is of course possible:

  php-worker:
    build:
      context: .
      target: frankenphp_dev
    image: ${IMAGES_PREFIX:-}app-php
    restart: unless-stopped
    command: php bin/console messenger:consume async -v
    environment:
       - RUN_MIGRATIONS=false

And then react to it inside your entry point script:

if [ "$RUN_MIGRATIONS" = "true" ]; then

But I personally would advise against it.
At the moment you just have a simple php container that is no different from the others, except for the command.
That's simple and easy.

Since migrations cannot be executed twice anyway there is no need for additional logic imho.
But it's up to you.

[norkunas]

Thanks :)

[norkunas]

Since migrations cannot be executed twice anyway there is no need for additional logic imho.

Yes, but it's slows down a little bit container startup time :)

[noximo]

Can I ask why the target is frankenphp_dev and not prod?
Or it should be overridden in compose.prod.yml?

[norkunas]

For the prod I run the build in github actions and then fetch from ghcr registry, but if you want to build on your server then I think the answer is yes, it should be overriden in compose.prod.yaml

[noximo]

Thank you!

[noximo]

The configuration didn't work for me in the end and I had to do some tweaks.

In compose.yml:

services:
    php:
        image: ${IMAGES_PREFIX:-}app-php
        restart: unless-stopped
        environment:
            ...
            RUN_MIGRATIONS: true
        volumes:
            - ...
            - var_log:/app/var/log
    php-worker:
        build:
            context: .
            target: frankenphp_dev
        image: ${IMAGES_PREFIX:-}app-php
        restart: unless-stopped
        command: php bin/console messenger:consume -vvv
        environment:
            DATABASE_URL: postgresql://${POSTGRES_USER:-app}:${POSTGRES_PASSWORD:-!ChangeMe!}@database:5432/${POSTGRES_DB:-app}?serverVersion=${POSTGRES_VERSION:-15}&charset=${POSTGRES_CHARSET:-utf8}
            RUN_MIGRATIONS: false
        volumes:
            - var_log:/app/var/log
        depends_on:
            - php
            - database

in compose.override.yml:

services:
    php:
        ....
    php-worker:
        volumes:
            - ./:/app

and finally compose.prod.yml:

services:
    php:
        build:
            context: .
            target: frankenphp_prod
        environment:
            APP_SECRET: ${APP_SECRET}
        volumes:
            - var_log:/app/var/log
    php-worker-async:
        build:
            context: .
            target: frankenphp_prod
        environment:
            APP_SECRET: ${APP_SECRET}
        volumes:
            - var_log:/app/var/log

Most important is the volume in compose.override.yml. Without it, worker had no access to code and kept trying to install symfony from scratch. (Which wasn't problem when targeting frankenphp_prod as that contains all the linking in Dockerfile directly).
The var_log volume is something I need, without it I wasn't able to access logs from worker within the main app.

I'm not that versed in docker, so this may not the be optimal solution but it works for me so far. Nonetheless, if there's something to improve, I'm all ears.

[noximo]

Another improvement. I need to run several consumers in parallel. I guess that would be a good use case for the supervisor but I was unable to get that running, so I circled back to docker compose, despite some of its drawbacks.

The config is mostly the same as in the above examples, I just specified transport in the command.

Then I've added this to compose.yml:

    php-worker-slow:
        extends:
            service: php-worker
        command: php bin/console messenger:consume scheduler_slow -vvv --time-limit=60 --limit=10 --memory-limit=128M
    php-worker-fast:
        extends:
            service: php-worker
        command: php bin/console messenger:consume scheduler_fast -vvv --time-limit=60 --limit=10 --memory-limit=128M

and this to the other two:

  php-worker-slow:
        extends:
            service: php-worker
    php-worker-fast:
        extends:
            service: php-worker

Running more instances in parallel should be just a matter of duplicating the service under a new name. Sadly, you need to duplicate in all three files

[Moskito89]

Another improvement. I need to run several consumers in parallel. I guess that would be a good use case for the supervisor but I was unable to get that running, so I circled back to docker compose, despite some of its drawbacks.

The config is mostly the same as in the above examples, I just specified transport in the command.

Then I've added this to compose.yml:

    php-worker-slow:
        extends:
            service: php-worker
        command: php bin/console messenger:consume scheduler_slow -vvv --time-limit=60 --limit=10 --memory-limit=128M
    php-worker-fast:
        extends:
            service: php-worker
        command: php bin/console messenger:consume scheduler_fast -vvv --time-limit=60 --limit=10 --memory-limit=128M

and this to the other two:

  php-worker-slow:
        extends:
            service: php-worker
    php-worker-fast:
        extends:
            service: php-worker

Running more instances in parallel should be just a matter of duplicating the service under a new name. Sadly, you need to duplicate in all three files

I took another way and used the --scale argument, what also worked well. Following your example, @noximo, running the php-worker 3 times would look like that:

docker compose up --pull always -d --wait --scale php-worker=3

[noximo]

Ah, neat. I'll still need to keep it in separate consumers because I don't want long-running tasks blocking the short ones, but at least I won't need to duplicate workers with the same transport.

[gremo]

Customize the frankenphp image and install supervisor to run Symfony messenger.

[7-zete-7]

@gremo, using supervisor inside of Docker is a bad practice (see https://stackoverflow.com/a/65570526/5533907, https://docs.docker.com/engine/containers/multi-service_container/).

Best way to start Symfony Messenger consumer for this image is just change the command parameter of default php service configuration (see examples above) with adding pcntl PHP extension (to handle stop-signals in right way).

[gremo]

@7-zete-7 In general, I agree, but here we are talking about a small system utility (like cron). Everything would still be logged by Symfony and, in addition, it's not necessary to "replicate" the environemnt in a new container just to run the command.

[7-zete-7]

@gremo, starting multiple processes inside of single container have many bad consequences. In case of Symfony Messenger you loose parallel executing (when you can start multiple instances of service). In case of Docker container you have issue of message handler interaption when php container be restarted. Incase of Symfony Scheduler you need to do more configuration to have stable working scheduler.

[gremo]

@7-zete-7 thanks for your point of view!

It would be very nice indeed to create a quick "how to" to make supervisor and/or cron work.

[VerlooveMaxime]

I'm starting to get a little bit confused about this topic. I'm currently running the consume command in a separate container (php-worker), but would it be possible to run the consume command in the same default php container? Which option is the better choice?

Another remark when using two containers: the dockerfile has the caddy health check baked in, so the php-worker container is being reported as unhealthy.

[n3o77]

The idea with docker is that each container has a single responsibility. So if that service needs more resources you just start more containers to scale up.

Depending on your needs you could use supervisord to run all consumers in a single container as described here: https://docs.docker.com/engine/containers/multi-service_container/#use-a-process-manager or for crons something like this: https://stackoverflow.com/questions/37458287/how-to-run-a-cron-job-inside-a-docker-container/37458519#37458519 . Either container would need to be build up upon the php container so your PHP config etc. is available.

The approach from @noximo works quite well. If you want to change the healthcheck for the workers you can install procps in the dockerfile https://github.com/dunglas/symfony-docker/blob/main/Dockerfile#L24 and then use something like this:

  php-worker:
    scale: 2
    extends:
      service: php
    healthcheck:
      test: ["CMD-SHELL", "/usr/bin/pgrep php || exit 1"]
      interval: 3s
      start_period: 10s
    command: php bin/console messenger:consume async --time-limit=60

[7-zete-7]

Well said, @n3o77.

But seems like this healthcheck have no benefits. The php process will be started immediately when the container starts, and the container will be automatically stopped when the process stops; if the process hangs, this healthcheck will be a false-positive.

It would be nice if Symfony had a built-in mechanism for checking the health of the consumer. In most cases, it makes sense to simply disable health checking for this service.

The presence of the --limit and --time-limit parameters in the command should be enough to be able to monitor whether this service is working correctly (regularly low uptime - there is an initialization problem or the --limit parameter value is too small; too high uptime - the process is hangs or the --time-limit parameter value is too high).

As an alternative to CRON within the application, Symfony has a (relatively new) Scheduler component that allows you to run scheduled and/or regular tasks within the Messenger component. The most important thing when working with this component is to remember to configure it for a production (see https://symfony.com/doc/current/scheduler.html#efficient-management-with-symfony-scheduler).

An alternative to CRON within the infrastructure can be either something built-in (independent of applications). As an option for such a solution that does not require changing the application and/or its images (it was very helpful in such cases) — Ofelia - a job scheduler.

[gremo]

Thank you @7-zete-7 for your usefull tips!

I was able to run supervisord as you said, in a separate php worker container, with a few modifications:

    # compose.yaml
    php-worker:
        extends:
            service: php
        container_name: ${COMPOSE_PROJECT_NAME}-php-worker
        command: ['bash', '-c', '/usr/bin/supervisord -c /etc/supervisor/supervisord.conf']
        restart: unless-stopped

   # compose.dev.yaml
    php-worker:
        extends:
            service: php
        volumes:
            - ./config/docker/supervisor.conf:/etc/supervisor/conf.d/app.conf

In the Dockerfile:

    apt-get -y --no-install-recommends install supervisor; \
    sed -i '/\[supervisord\]/a user=root' /etc/supervisor/supervisord.conf; \
    sed -i '/\[supervisord\]/a nodaemon=true' /etc/supervisor/supervisord.conf; \

supervisorctl status
messenger-consume:messenger-consume_00   RUNNING   pid 1126, uptime 0:00:00
messenger-consume:messenger-consume_01   RUNNING   pid 1122, uptime 0:00:00

The only thing I'm trying to solve with this configuration is the fact that, even rebuilding the container, the worker seems to "remember" the old configuration.

For cron, it's quite the same concept. I'm using Messenger component so I don't need cron, I'll go for the Scheduler component!

[7-zete-7]

Great example, @gremo! This is really useful information.

However, all this can be achieved without using Supervisor. This will allow you to not complicate the configuration and not create a service that is not controlled by Docker.

The same configuration can be written as follows:

services:
  ...
  consumer:
    extends: php
    restart: unless-stopped
    deploy:
      replicas: 2
    healthcheck:
      disable: true
      #command: /usr/bin/true # if "disable: true" isn't works

    # if you need to disable the entry point provided by this repository
    #entrypoint: ""

    # https://symfony.com/doc/current/messenger.html#deploying-to-production
    command: bin/console messenger:consume --limit=... --time-limit=... async ...

With this configuration, no additional packages need to be installed (except the pcntl PHP extension). The work of the processes will be controlled and monitored by Docker. That is, it will be possible to manage the number of replicas, stop and start processes without additional manipulations with the supervisor configuration.

An example of changing the number of running processes using this service configuration:

# change replicas count to 6
docker compose up --detach --scale consumer=6 --no-deps consumer

# restart all messenger:consume processes
docker compose restart consumer

The Symfony documentation talks about using Process Manager, but it's talking about a non-Docker environment.

[VerlooveMaxime]

Thanks for your input guys. Turns out I was doing everything the proper way already, just need to change the health check. Seems the best option for now is to just disable it for the consumer.

[gremo]

@7-zete-7 Thank you again! I'm going to test Supervisor in production. I’m not sure, but putting a Symfony command directly in the Compose files feels a bit odd to me. I guess it's just some programmer paranoia. 😄

For the healthcheck:

        healthcheck:
            test: ['CMD', 'supervisorctl', 'status']
            interval: 30s
            timeout: 5s
            retries: 3

[maxhelias]

Feel free to watch this PR and give us your feedback: #681