Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

index out of range at HexPatriciaHashed. unfoldBranchNode on Eth Mainnet Fresh sync #13716

Open
taratorio opened this issue Feb 6, 2025 · 7 comments
Open

index out of range at HexPatriciaHashed. unfoldBranchNode on Eth Mainnet Fresh sync #13716

taratorio opened this issue Feb 6, 2025 · 7 comments
Assignees
@awskii
Labels
erigon3 imp0 Critical importance

Comments

@taratorio
Copy link
Member

System information

Erigon version: git_branch=main git_tag=v3.0.0-beta1-107-gc948a59 git_commit=c948a59ced202cd12d59c96e8293fc7e0c13490f

OS & Version: Linux

Commit hash: c948a59

Erigon Command (with flags/config):

build/bin/erigon \
--chain=mainnet \
--datadir=/home/erigon/eth/datadir \
--torrent.download.rate=10G --torrent.download.slots=400 \
--metrics \
--pprof \
--http.addr=0.0.0.0 \
--private.api.addr=0.0.0.0:29090 \
--http.port=28545 \
--authrpc.port=28551 \
--ws.port=28546 \
--torrent.port=62069 \
--port=50303 \
--p2p.allowed-ports=50303,50304,50305,50306,50307 \
--caplin.discovery.port=24000 \
--caplin.discovery.tcpport=24001 \
--sentinel.port=27777 \
--beacon.api.port=25555 \
--shutter.p2p.listen.port=43102 \
--pprof.port=26060 \
--metrics.port=26061 \
--diagnostics.endpoint.port=26062 \
--log.dir.verbosity=debug

Consensus Layer: Caplin (internal by default)

Chain/Network: Ethereum Mainnet

Expected behaviour

fresh sync should finish without issues

Actual behaviour

[INFO] [02-05|19:10:28.196] [4/6 Execution]                          blk=21781678 blks=372 blk/s=18.6 txs=74807 tx/s=3.74k gas/s=338.33M buf=56.3MB/512.0MB stepsInDB=1.00 step=1745.1 inMem=false alloc=8.0GB sys=25.7GB
[INFO] [02-05|19:10:48.204] [4/6 Execution]                          blk=21782086 blks=409 blk/s=20.4 txs=77808 tx/s=3.88k gas/s=370.28M buf=83.9MB/512.0MB stepsInDB=1.00 step=1745.1 inMem=false alloc=8.1GB sys=25.7GB
[INFO] [02-05|19:10:57.636] [4/6 Execution] Done                     blk=21782290 blks=1310 blk/s=18.9 txs=255341 tx/s=3.67k gas/s=341.39M buf=95.7MB/512.0MB stepsInDB=0.00 step=1745.2 inMem=false alloc=10.4GB sys=25.7GB
panic: runtime error: index out of range [104] with length 104

goroutine 7602055 [running]:
github.com/erigontech/erigon-lib/commitment.(*HexPatriciaHashed).unfoldBranchNode(0xc31829e000, 0x5, 0x6, 0x0)
        github.com/erigontech/[email protected]/commitment/hex_patricia_hashed.go:1353 +0x6e6
github.com/erigontech/erigon-lib/commitment.(*HexPatriciaHashed).unfold(0xc31829e000, {0xc1876846c0?, 0x40?, 0x40?}, 0x1)
        github.com/erigontech/[email protected]/commitment/hex_patricia_hashed.go:1410 +0x567
github.com/erigontech/erigon-lib/commitment.(*HexPatriciaHashed).Process.func1({0xc1876846c0, 0x40, 0x40}, {0xc18bfb5158, 0x14, 0x14}, 0x0)
        github.com/erigontech/[email protected]/commitment/hex_patricia_hashed.go:2038 +0x605
github.com/erigontech/erigon-lib/commitment.(*Updates).HashSort.func1({0xc1876846c0?, 0x0?, 0xc00b163808?}, {0xc18bfb5158?, 0xc00b163808?, 0xbb1a33?}, {0xc00b163910?, 0x0?}, 0x1?)
        github.com/erigontech/[email protected]/commitment/commitment.go:1068 +0x25
github.com/erigontech/erigon-lib/etl.(*Collector).Load.func3({0xc1876846c0?, 0xc187684640?, 0x0?}, {0xc18bfb5158?, 0xc18bfb5140?, 0x0?})
        github.com/erigontech/[email protected]/etl/collector.go:267 +0x31
github.com/erigontech/erigon-lib/etl.mergeSortFiles({0x2c67927, 0xa}, {0xc219889b00, 0x2, 0x2?}, 0xc00b163ad0, {0xc000fea070, 0x0, 0x0, {0x0, ...}, ...}, ...)
        github.com/erigontech/[email protected]/etl/collector.go:353 +0x752
github.com/erigontech/erigon-lib/etl.(*Collector).Load(0xc0f102a480, {0x0, 0x0}, {0x0, 0x0}, 0xc00d9c75f0, {0xc000fea070, 0x0, 0x0, {0x0, ...}, ...})
        github.com/erigontech/[email protected]/etl/collector.go:269 +0x6eb
github.com/erigontech/erigon-lib/commitment.(*Updates).HashSort(0xc15f176780, {0x34ffa10, 0xc00128dc20}, 0xc106297d10)
        github.com/erigontech/[email protected]/commitment/commitment.go:1067 +0x1d3
github.com/erigontech/erigon-lib/commitment.(*HexPatriciaHashed).Process(0xc31829e000, {0x34ffa10, 0xc00128dc20}, 0xc15f176780, {0xc00368af40, 0xd})
        github.com/erigontech/[email protected]/commitment/hex_patricia_hashed.go:2016 +0x272
github.com/erigontech/erigon-lib/state.(*SharedDomainsCommitmentContext).ComputeCommitment(0xc15f176730, {0x34ffa10, 0xc00128dc20}, 0x1, 0x14c5f12, {0xc00368af40, 0xd})
        github.com/erigontech/[email protected]/state/domain_shared.go:1354 +0x39c
github.com/erigontech/erigon-lib/state.(*SharedDomains).ComputeCommitment(...)
        github.com/erigontech/[email protected]/state/domain_shared.go:726
github.com/erigontech/erigon/eth/stagedsync.flushAndCheckCommitmentV3({_, _}, _, {_, _}, _, {{0x35184f0, 0xc0042900a8}, 0x20000000, {0x1, ...}, ...}, ...)
        github.com/erigontech/erigon/eth/stagedsync/exec3.go:862 +0x13a
github.com/erigontech/erigon/eth/stagedsync.ExecV3({_, _}, _, {_, _}, _, {{0x35184f0, 0xc0042900a8}, 0x20000000, {0x1, ...}, ...}, ...)
        github.com/erigontech/erigon/eth/stagedsync/exec3.go:724 +0x2865
github.com/erigontech/erigon/eth/stagedsync.ExecBlockV3(_, {_, _}, {{_, _}, {_, _}, _}, _, {0x34ffa10, ...}, ...)
        github.com/erigontech/erigon/eth/stagedsync/stage_execute.go:162 +0x20f
github.com/erigontech/erigon/eth/stagedsync.SpawnExecuteBlocksStage(_, {_, _}, {{_, _}, {_, _}, _}, _, {0x34ffa10, ...}, ...)
        github.com/erigontech/erigon/eth/stagedsync/stage_execute.go:256 +0x10e
github.com/erigontech/erigon/eth/stagedsync.PipelineStages.func10(0x9?, 0x7bcfd085f880?, {0x34f9210?, 0xc001c834a0?}, {{0x3534d80, 0xc34a180800}, {0x0, 0x0}, 0x0}, {0x3516c48, ...})
        github.com/erigontech/erigon/eth/stagedsync/default_stages.go:238 +0xf0
github.com/erigontech/erigon/eth/stagedsync.(*Sync).runStage(0xc001c834a0, 0xc003249400, {0x35184f0, 0xc0042900a8}, {{0x3534d80, 0xc34a180800}, {0x0, 0x0}, 0x0}, 0x0, ...)
        github.com/erigontech/erigon/eth/stagedsync/sync.go:528 +0x190
github.com/erigontech/erigon/eth/stagedsync.(*Sync).Run(0xc001c834a0, {0x35184f0, 0xc0042900a8}, {{0x3534d80, 0xc34a180800}, {0x0, 0x0}, 0x0}, 0xf6?, 0x0)
        github.com/erigontech/erigon/eth/stagedsync/sync.go:412 +0x2ad
github.com/erigontech/erigon/turbo/execution/eth1.(*EthereumExecutionModule).updateForkChoice(0xc000b072c0, {0x34ffa10, 0xc00128dc20}, {0x33, 0x4a, 0x33, 0xa1, 0x50, 0x38, 0x15, ...}, ...)
        github.com/erigontech/erigon/turbo/execution/eth1/forkchoice.go:449 +0x1527
created by github.com/erigontech/erigon/turbo/execution/eth1.(*EthereumExecutionModule).UpdateForkChoice in goroutine 7599958
        github.com/erigontech/erigon/turbo/execution/eth1/forkchoice.go:130 +0x31c

Steps to reproduce the behaviour

Run a fresh sync from scratch on eth mainnet
@taratorio taratorio added erigon3 imp1 High importance labels Feb 6, 2025
@taratorio taratorio added this to the 3.0.0-beta2 milestone Feb 6, 2025
@yperbasis yperbasis added imp0 Critical importance and removed imp1 High importance labels Feb 6, 2025
@awskii
Copy link
Member

awskii commented Feb 7, 2025

can you start with --downloader.verify?

@taratorio
Copy link
Member Author

@awskii you can run this on your machine and reproduce

@jquave
Copy link

jquave commented Feb 8, 2025

Getting what seems to be the same issue on a new eth node sync. I also tried with the --downloader.verify arg and it still produced the same problem. I also after experiencing this checked out v3.0.0-beta1@0b94461 and rebuilt. This resulted in the same outcome, I'm now wondering if I have to completely delete the 1TB of stuff that's been downloaded to make this work. Any other things to try?

INFO[02-08|15:36:46.902] [4/6 Execution] Done                     blk=21789312 blks=3314 bl
k/s=14.4 txs=662205 tx/s=2.88k gas/s=262.44M buf=230.0MB/512.0MB stepsInDB=0.00 step=1746.0
 inMem=false alloc=7.6GB sys=18.5GB
panic: runtime error: index out of range [36] with length 36

goroutine 2744945 [running]:
github.com/erigontech/erigon-lib/commitment.(*HexPatriciaHashed).unfoldBranchNode(0xc2e03d4
000, 0x6, 0x7, 0x0)
        github.com/erigontech/[email protected]/commitment/hex_
patricia_hashed.go:1346 +0x6e6
github.com/erigontech/erigon-lib/commitment.(*HexPatriciaHashed).unfold(0xc2e03d4000, {0xc2
77e659c0?, 0x40?, 0x40?}, 0x1)
        github.com/erigontech/[email protected]/commitment/hex_
patricia_hashed.go:1403 +0x567
github.com/erigontech/erigon-lib/commitment.(*HexPatriciaHashed).Process.func1({0xc277e659c
0, 0x40, 0x40}, {0xc35527b5c0, 0x14, 0x14}, 0x0)
        github.com/erigontech/[email protected]/commitment/hex_
patricia_hashed.go:2036 +0x605
github.com/erigontech/erigon-lib/commitment.(*Updates).HashSort.func1({0xc277e659c0?, 0x0?,
 0x0?}, {0xc35527b5c0?, 0xc219fdba18?, 0xb8f5f3?}, {0xc219fdbb20?, 0x0?}, 0x2?)
        github.com/erigontech/[email protected]/commitment/comm
itment.go:1126 +0x25
github.com/erigontech/erigon-lib/etl.(*Collector).Load.func3({0xc277e659c0?, 0xc277e65a00?,
 0x0?}, {0xc35527b5c0?, 0xc35527b5d8?, 0x0?})
        github.com/erigontech/[email protected]/etl/collector.g
o:267 +0x31

@AskAlexSharov
Copy link
Collaborator

Switch to latest “main”
rm -rf datadir/chaindata

@jquave
Copy link

jquave commented Feb 9, 2025

Switch to latest “main” rm -rf datadir/chaindata

same behavior

@AskAlexSharov
Copy link
Collaborator

Then:
Switch to latest “main”
erigon snapshots rm-state-snapshots —step=latest —datadir=your
rm -rf datadir/chaindata

But I would advise: re-sync.
Because resync will fix couple other bugs also.

@taratorio
Copy link
Member Author

working fine after we reverted 1f87c16 in 3e9aea6

@awskii awskii removed this from the 3.0.0-beta2 milestone Feb 10, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@awskii awskii

Labels
erigon3 imp0 Critical importance
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
5 participants
@jquave @awskii @yperbasis @AskAlexSharov @taratorio