Benchmark L1 info tree sync

[arnaubennassar]

Investigate why is this so slow (chekcing against real env such as Bali)

• Fix the slowness (chekcing against real env such as Bali)
• Ideally test against multiple providers
• Add a benchmark using the simulated env, corroborate that the results make sense vs the real env. Or add a test that uses a real env and introduce a GH secret to connect to a real RPC, have a deathline on the test

 

more details here

[vcastellm]

l1infotreesync [[Benchmark]] for [[CDK 2.0]]

After initial investigation using cpu profiling with a setting of 2000 block range against Bali network fork RPC in [[Tenderly]] majority of time is spent in the Tree implementation, using SQL as a storage backend for the tree https://flamegraph.com/share/c8814381-b6dd-11ef-8d9a-2e2d4d104be7

go test -timeout 240s -cpuprofile cpu4.out -run ^TestBenchBali$ github.com/0xPolygon/cdk/l1infotreesync -v

Going to try with a higher block range it doesn't work with 500k range

• rpc.jsonError=&{-32602 invalid params Query returned more than 20000 results. Try with this block range [0x565539, 0x5c93ef]
• Now trying with 100k https://flamegraph.com/share/adaa7156-b6e6-11ef-9230-a2b3402f0dcf

The Flamegraphs reveled that there's a bottleneck in the DB access for the tree and in the JSON decoding of the reponses.

We're gonna tackle both issues, maybe using https://github.com/valyala/fastjson and trying to optimize the DB setup.

I've tried several configs for [[SQLite3]] so far:

• Default journal mode ROLLBACK https://flamegraph.com/share/77c313cc-b71a-11ef-b027-3a58f36d25ce
• WAL Journal clearly improves the DB performance https://flamegraph.com/share/9f0d0e90-b71a-11ef-b027-3a58f36d25ce
• MEMORY journal seems like a medium term https://flamegraph.com/share/e3a0589f-b71b-11ef-9832-26c3e5347170
  The recommended setting should be WAL in most of the cases
  Noticed a reduction in Database query time, I'm considering this as optimized at this point.

We'll use ethgo to try to optimize blockheader query in the evmdownloader.go in the sync package

Using ethgo is showing that it has worst performance than go-ethereum https://flamegraph.com/share/173a1166-bbc9-11ef-9832-26c3e5347170

I'll try another path:

• Querying every block header is inefficient, let's check only the last one, usually it will just work because the won't be any reorg in normal conditions, when there's a reorg, all the work will be discarded and then re-processed but it should be much more efficient in general case.