Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve debug_traceBlock calls performance #8103

Open
wants to merge 16 commits into
base: main
Choose a base branch
from

Conversation

ahamlat
Copy link
Contributor

@ahamlat ahamlat commented Jan 10, 2025

PR description

This PR is the same as this PR #8076, I closed by error.

It introduces several key improvements to the debug_traceBlock, debug_traceBlockByNumber, and debug_traceBlockByHash methods, significantly enhancing both performance and memory usage:

The core change is the adoption of a pipeline architecture, where:

One thread handles transaction execution.
Other threads processes the execution results and generates the JSON output. This parallelization ensures better resource utilization and reduces overall execution time.
Memory usage is optimized by the pipeline processing approach. Instead of retaining all trace frames for every transaction, the pipeline now keeps only the trace frames relevant to the current transaction, significantly reducing memory consumption.

The transformation of trace frames into Java JSON representations now occurs in parallel.

Default trace options have been adjusted to align with the behavior of Geth and Nethermind:

disableStorage is set to false (enabling storage tracing).
disableMemory is set to true (disabling memory tracing).
disableStack is set to false (enabling stack tracing).
The structure of the JSON output has been modified to match Geth's format:

Hexadecimal values are now displayed as short hex strings (e.g., 0x1, 0xabc), which is more compact and efficient.
The stack array is always displayed when the option is enabled, even when empty, to ensure consistency with other Ethereum clients. However, memory and storage are omitted if they are empty, which reduces unnecessary data.
Storage keys and values are now displayed without the 0x prefix, in line with Geth's behavior.
Change the implementation of toShortHexString, by creating a new method toCompactHex. The main improvement is to not relay on Bytes, but on byte[] to avoid dynamic dispatch mechanism related to the use of methods get(i) and size(). Those methods are not used anymore thanks to the switch to StringBuilder, even if it has its own cons. This method will replace in the future toFastHex implementation, once we have finished the work on Tuweni.

These changes have resulted in a 3X performance improvement, reducing the size of the output by 2 to 3 times, making previously unresponsive calls fast enough with comparable performance with other clients. The optimizations not only improve response times but also allow for smoother operation with lower memory usage, even when processing large traces.

Overall, the new approach ensures better performance, reduced resource usage, and more consistent output with other Ethereum clients, particularly Geth and Nethermind.

Fixed Issue(s)

Fixes #5322

Thanks for sending a pull request! Have you done the following?

  • Checked out our contribution guidelines?
  • Considered documentation and added the doc-change-required label to this PR if updates are required.
  • Considered the changelog and included an update if required.
  • For database changes (e.g. KeyValueSegmentIdentifier) considered compatibility and performed forwards and backwards compatibility tests

Locally, you can run these tests to catch failures early:

  • unit tests: ./gradlew build
  • acceptance tests: ./gradlew acceptanceTest
  • integration tests: ./gradlew integrationTest
  • reference tests: ./gradlew ethereum:referenceTests:referenceTests

ahamlat added 15 commits January 2, 2025 17:43
- Change debug_traceBlockByNumber implementation by using a pipeline
- Change the defaults to match Geth and Nethermind defaults
- Improve general performance
- Add unit tests

Signed-off-by: Ameziane H. <[email protected]>
…egration test because they were not relevant.

Signed-off-by: Ameziane H. <[email protected]>
Signed-off-by: Ameziane H. <[email protected]>
Signed-off-by: Ameziane H. <[email protected]>
Signed-off-by: Ameziane H. <[email protected]>
from Besu Controller.

Signed-off-by: Ameziane H. <[email protected]>
Signed-off-by: Ameziane H. <[email protected]>
Signed-off-by: Ameziane H. <[email protected]>
@ahamlat ahamlat changed the title Debug trace with pipeline Improve debug_traceBlock calls performance Jan 10, 2025
@matkt matkt force-pushed the debug-trace-with-pipeline branch from c00f7db to 447907b Compare January 10, 2025 14:41
@ahamlat ahamlat marked this pull request as ready for review January 10, 2025 14:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

implement pipeline on debug_ tracing endpoints
1 participant