Initial draft of local test framework RFC #8896
Conversation
There was a problem hiding this comment.
This all looks great, with the exception of the approach for streaming logs back to the test executive. I think we need to think of a more robust solution upfront for this, since our goal here is to run local networks with snarks turned off (see my other comment for details of why this is relevant).
rfcs/0042-local-test-framework.md
Outdated
|
|
||
| # Unresolved Questions | ||
|
|
||
| - Is compiling a docker-compose file the right approach for scheduling the containers? The nice thing about using a docker-compose file is that all network management should be automatic. |
There was a problem hiding this comment.
Yeah, I think this is the right way to go, especially for the network management bit.
rfcs/0042-local-test-framework.md
Outdated
|
|
||
| - Is using a different service for each type of node the best effective approach? Would it be better to launch all nodes under the same service in the docker-compose file? | ||
|
|
||
| - Is polling each service and then aggregating those logs the best approach? Would it be better to do filtering before aggregating? |
There was a problem hiding this comment.
I think the polling logs in the local integration test engine is not the correct approach.
We do a pull-based approach for the cloud engine, and that works out fine for us right now because we pre-filter the logs (so the volume we handle per poll is very low), but more importantly, because we are only running networks with snarks enabled. When snarks are enabled, the networks move pretty slow, so small delays in the order of 5-10 seconds are not a problem.
However, with the local engine, the goal is the run snarkless networks, which move extremely quickly by comparison. In that world, a 5-10 second delay in logs could break the test. Furthermore, the log volume per second of a snarkless network is massive by comparison due to the speed the network operates at. Because of this, I think we both need to pre-filter the logs in the docker containers (to bring the log volume the single-threaded test_executive needs to handle down to a fraction of the size), and we need to be reading those logs off of a pipe in a more push-based fashion.
Another advantage of using a push-based pipe approach here is that you don't need to deal with re-reading logs you already handled, which appears to be a problem you would need to solve with the docker swarm logging interface. You might be able to get by with asking docker swarm to only give you logs that were recorded since the last time you asked for logs, but you have some risk of missing logs in that approach and are relying on docker swarm's performance to quickly perform that query on it's log cache.
There was a problem hiding this comment.
Followed up with a restructuring of the proposed polling solution to use the push-based pipe approach. 👍
rfcs/0042-local-test-framework.md
Outdated
|
|
||
| - Is polling each service and then aggregating those logs the best approach? Would it be better to do filtering before aggregating? | ||
|
|
||
| - Does this plan capture the overall direction we want the local testing framework to go? |
|
|
||
| The new local testing engine must implement all existing features which include: | ||
|
|
||
| - Starting/Stopping nodes dynamically |
There was a problem hiding this comment.
Just wanted to mention as a note here: If you continue using the puppeteer container (which I recommend), then this is very easy to do. You can peek at how this works in the cloud engine, but starting/stopping a puppeteered node is as simple as running a shell script in the container which communicates with the puppeteer process.
A good point was made about restructuring the proposed logging architecture. Instead of polling on an interval with Docker CLI which can lead to latency issues, we can redirect all stdout to a pipe shared with the test executive. This change reflects the following changes in the rfc to address that feedback.
|
Moved too o1-labs/rfcs#40 |
Proposes a containerization-focused solution on the architecture and overall workflow of the local test framework.