Lucy Local Framework extension

[MartinMinkov]

Summary

Implements o1-labs/rfcs#40 🗒️

This PR implements the local extension of Lucy to use Docker Swarm to run integration tests on a single host. As extensive testing confirms, all existing integration tests run successfully on the local version. The work to add this to CI will not be added in this PR, but in a follow-up PR where we can spin up a single machine and run the tests locally.

Changes from the RFC

This PR changes implementation details outlined in the RFC regarding node communication between Lucy and how logs are gathered. Inside the RFC, the original plan was to mount a Unix pipe to ingest all logs from the container, which the Lucy process would read from and process events for test conditions. Instead of that approach, this PR implements the same log-gathering process as the cloud integration, in which we utilize the GraphQL interface of each node in an integration test.

For more details, the cloud integration uses a GraphQL interface to tell nodes to start collecting structured logs and expose those structured logs via a GraphQL query. When nodes are first launched, Lucy will make a StartFilteredLog mutation, which tells the node to start collecting any structured logs it generates. Then, we can query for those structured logs and use them as test conditions with the same GraphQL interface.

This approach is cleaner than the original pipe-based approach because we can leverage the same architecture for both backends. Additionally, it makes communication between nodes easier to handle, as mounting a pipe and reading logs can have performance issues and blocked logs. Additionally, the pipe-based approach does not let us scale to multiple Docker Swarm nodes if we want to extend the functionality of this backend to use additional machines outside of just one host.

So, in summary, we do not use a pipe-based approach for logs since we have an existing infrastructure that we can use via GraphQL polling that works excellently. It can be extended further if we want to add additional features.

Details on machine performance

I run an AMD Ryzen 9 5900X 12-Core Processor on my machine and have 64GB of RAM. While running tests, the most significant strain on my machine is done in the bootstrap phase when nodes initialize. While a node is initializing, it uses about 6 CPU threads and around 4-5GB of memory, and then resource strain flattens once the node is synced. The next performance strain is also from SNARK workers, which take up 6-8 CPU threads. I haven't had any issues with tests failing due to limited resources, even when spinning up 8+ nodes during integration tests. These performance strains are burst-oriented, meaning that when nodes initialize, or SNARK workers are doing their computing, a burst of resources is used but flatlines when that work is done.

Running the tests

If you would like to run an integration test, you can use the following script as an example:

#!/bin/bash
set -e
make build_intgtest 
MINA_IMAGE=gcr.io/o1labs-192920/mina-daemon:2.0.0rampup5-develop-1e360f7-bullseye-berkeley
MINA_ARCHIVE=gcr.io/o1labs-192920/mina-archive:2.0.0rampup5-develop-1e360f7-bullseye
TEST=zkapps
ulimit -n 65532
script -f -c "./_build/default/src/app/test_executive/test_executive.exe local $TEST --mina-image=$MINA_IMAGE --archive-image=$MINA_ARCHIVE --debug" test.log

[MartinMinkov]

I made the GraphQL polling log engine a functor so that both the cloud and the local integrations can utilize it. In addition to making it a functor, I added a parameter to specify an interval to issue GraphQL requests to start collecting events, which is vitally important for the local version.

It's important because Lucy will wait for all nodes to initialize as a test condition. These conditions are checked by issuing GraphQL requests to the nodes to start collecting their events and make them available in a query. There is a race condition where a node can initialize before it's been set to start collecting it's logs, which means the node initialization event will be missed and the tests fail. To fix this, we make GraphQL requests much more frequent on the local version to make nodes immediately start collecting logs after the GraphQL endpoint is available.

I thought about making the GraphQL availability it's own structured event which we listen too, so that we can avoid this interval parameter and just issue a request to collect events right when GraphQL is available. I decided not to do that since we would need to land that change into different branches so that the daemon emits that event, use newly built Docker images instead of any old ones for testing, and I figured this is an easier way to unblock this issue.

The local version issues requests every 0.25 seconds to nodes and the cloud version keeps the same 10 seconds. This is just to kick off the event collection, normal polling queries still use the same time as before 10 seconds.

[MartinMinkov]

I define custom entry points for the daemon, postgres, and archive node containers. Would these be better served in a directory and removed from the inline definition? Where is the best location for these to live if we wanted to move them?

[stevenplatt]

Viewing the infra code, the current standard appears to be to extract this and move it to mina/dockerfiles/puppeteer-context/.

Our src dockerfiles would then be updated to include the files within the docker container image (example can be seen here).

Those files inside the container image could then be referenced by the integration test engine by passing the --entrypoint value with the docker run command.

If I understand correctly, a version of this can be seen in kubernetes_network.ml line 117.

For now, I would leave the format as you have it (to avoid being blocked) and add an issue to the platform team zenhub, to have the additional entrypoints included in the standard docker builds.

(Edit: If you are familiar with the docker build flow in CI, you can attempt adding it as well. Myself or others in the platform team can pair with you to get it shipped, but I am making an assumption this would slow down other ITN/hardfork work and may not be helpful at the moment.)

[stevenplatt]

Viewing the infra code, the current standard appears to be to extract this and move it to mina/dockerfiles/puppeteer-context/.

Our src dockerfiles would then be updated to include the files within the docker container image (example can be seen here).

Those files inside the container image could then be referenced by the integration test engine by passing the --entrypoint value with the docker run command.

If I understand correctly, a version of this can be seen in kubernetes_network.ml line 117.

For now, I would leave the format as you have it (to avoid being blocked) and add an issue to the platform team zenhub, to have the additional entrypoints included in the standard docker builds.

(Edit: If you are familiar with the docker build flow in CI, you can attempt adding it as well. Myself or others in the platform team can pair with you to get it shipped, but I am making an assumption this would slow down other ITN/hardfork work and may not be helpful at the moment.)

[stevenplatt]

!ci-build-me

[dkijania]

!ci-build-me