Skip to content

Commit

Permalink
Remove old portal_debug API for seeding data into the network (#2726)
Browse files Browse the repository at this point in the history 
This was the first API created for this, it has been superseded
by the API that makes use of era1 files and/or the portal_bridge.

Only usage was still in test_portal_testnet which has now been
altered to make use if API calls from Portal specification.
  • Loading branch information
@kdeme
kdeme authored Oct 10, 2024
1 parent 59dde39 commit a5f0b12
Show file tree
Hide file tree
Showing 6 changed files with 34 additions and 324 deletions.
86 changes: 15 additions & 71 deletions fluffy/docs/the_fluffy_book/docs/history-content-bridging.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,11 @@

## Seeding from content bridges

### Seeding history data with the `portal_bridge`
### Seeding history content with the `portal_bridge`

The `portal_bridge` requires `era1` files as source for the block content from before the merge.
It requires access to a full node with EL JSON-RPC API for seeding the latest (head of the chain) block content.
Any block content between the merge and the latest is currently not implemented, but will be implemented in the future by usage of `era` files as source.

#### Step 1: Run a Portal client

Expand Down Expand Up @@ -45,7 +49,7 @@ WEB3_URL="http://127.0.0.1:8548" # Replace with your provider.
./build/portal_bridge history --latest:true --backfill:true --audit:true --era1-dir:/somedir/era1/ --web3-url:${WEB3_URL}
```

### Seeding post-merge history data with the `beacon_lc_bridge`
### Seeding post-merge history content with the `beacon_lc_bridge`

The `beacon_lc_bridge` is more of a standalone bridge that does not require access to a full node with its EL JSON-RPC API. However it is also more limited in the functions it provides.
It will start with the consensus light client sync and follow beacon block gossip. Once it is synced, the execution payload of new beacon blocks will be extracted and injected in the Portal network as execution headers
Expand All @@ -72,82 +76,22 @@ TRUSTED_BLOCK_ROOT=0x12345678901234567890123456789012345678901234567890123456789
./build/beacon_lc_bridge --trusted-block-root=${TRUSTED_BLOCK_ROOT}
```

## From locally stored block data

### Building and seeding epoch accumulators

#### Step 1: Building the epoch accumulators
1. Set-up access to an Ethereum JSON-RPC endpoint (e.g. local geth instance)
that can serve the data.

2. Use the `eth_data_exporter` tool to download and store all block headers into
*.e2s files arranged per epoch (8192 blocks):

```bash
make eth_data_exporter

./build/eth_data_exporter history exportEpochHeaders --data-dir:"./user_data_dir/"
```

This will store all block headers up till the merge block into *.e2s files in
the assigned `--data-dir`.

3. Build the master accumulator and the epoch accumulators:

```bash
./build/eth_data_exporter history exportAccumulatorData --write-epoch-records --data-dir:"./user_data_dir/"
```
## Seeding directly from the fluffy client

#### Step 2: Seed the epoch accumulators into the Portal network
Run Fluffy and trigger the propagation of data with the
`portal_history_propagateEpochRecords` JSON-RPC API call:
This method currently only supports seeding block content from before the merge.
It uses `era1` files as source for the content.

1. Run Fluffy and enable `portal_debug` JSON-RPC API:
```bash
./build/fluffy --rpc --rpc-api:portal,portal_debug

# From another terminal
curl -s -X POST -H 'Content-Type: application/json' -d '{"jsonrpc":"2.0","id":"1","method":"portal_history_propagateEpochRecords","params":["./user_data_dir/"]}' http://localhost:8545 | jq
```

2. Trigger the seeding of the content with the `portal_debug_historyGossipHeaders` and `portal_debug_historyGossipBlockContent` JSON-RPC methods.
The first method will gossip in the block headers, the second method will gossip the block bodies and receipts. It is important to first trigger the gossip of the headers because these are required for the validation of the bodies and the receipts.

#### Step 3 (Optional): Verify that all epoch accumulators are available
Run Fluffy and run the `content_verifier` tool to verify that all epoch
accumulators are available on the history network:

Make sure you still have a fluffy instance running, if not run:
```bash
./build/fluffy --rpc --rpc-api:portal,portal_debug
```

Run the `content_verifier` tool and see if all epoch accumulators are found:
```bash
make content_verifier
./build/content_verifier
```

### Downloading & seeding block data

1. Set-up access to an Ethereum JSON-RPC endpoint (e.g. local geth instance)
that can serve the data.
2. Use the `eth_data_exporter` tool to download history data through the
JSON-RPC endpoint into the format which is suitable for reading data into
Fluffy client and propagating into the network:

```bash
make eth_data_exporter

./build/eth_data_exporter history exportBlockData--initial-block:1 --end-block:10 --data-dir:"/user_data_dir/"
```

This will store blocks 1 to 10 into a json file located at
`./user_data_dir/eth-history-data.json`.

3. Run Fluffy and trigger the propagation of data with the
`portal_debug_history_propagate` JSON-RPC API call:

```bash
./build/fluffy --rpc --rpc-api:portal,portal_debug
curl -s -X POST -H 'Content-Type: application/json' -d '{"jsonrpc":"2.0","id":"1",
"method":"portal_debug_historyGossipHeaders","params":["/somedir/era1/"]}' http://localhost:8545 | jq

# From another shell
curl -s -X POST -H 'Content-Type: application/json' -d '{"jsonrpc":"2.0","id":"1","method":"portal_debug_history_propagate","params":["./user_data_dir/eth-history-data.json"]}' http://localhost:8545 | jq
curl -s -X POST -H 'Content-Type: application/json' -d '{"jsonrpc":"2.0","id":"1","method":"portal_debug_historyGossipBlockContent","params":["/somedir/era1/"]}' http://localhost:8545 | jq
```
10 changes: 0 additions & 10 deletions fluffy/eth_data/history_data_json_store.nim
View file
Original file line number Diff line number Diff line change
Expand Up @@ -100,16 +100,6 @@ iterator blocks*(
else:
error "Failed reading block from block data", error = res.error

iterator blocksContent*(
blockData: BlockDataTable, verify = false
): (ContentId, seq[byte], seq[byte]) =
for b in blocks(blockData, verify):
for value in b:
if len(value[1]) > 0:
let ckBytes = history_content.encode(value[0])
let contentId = history_content.toContentId(ckBytes)
yield (contentId, ckBytes.asSeq(), value[1])

func readBlockHeader*(blockData: BlockData): Result[Header, string] =
var rlp =
try:
Expand Down
193 changes: 1 addition & 192 deletions fluffy/eth_data/history_data_seeding.nim
View file
Original file line number Diff line number Diff line change
Expand Up @@ -8,209 +8,18 @@
{.push raises: [].}

import
std/[strformat, os],
results,
chronos,
chronicles,
../network/wire/portal_protocol,
../network/history/
[history_content, history_network, validation/historical_hashes_accumulator],
"."/[era1, history_data_json_store, history_data_ssz_e2s]
"."/[era1, history_data_ssz_e2s]

from eth/common/eth_types_rlp import rlpHash

export results

### Helper calls to seed the local database and/or the network

proc historyStore*(
p: PortalProtocol, dataFile: string, verify = false
): Result[void, string] =
let blockData = ?readJsonType(dataFile, BlockDataTable)

for b in blocks(blockData, verify):
for value in b:
let encKey = history_content.encode(value[0])
# Note: This is the slowest part due to the hashing that takes place.
p.storeContent(encKey, history_content.toContentId(encKey), value[1])

ok()

proc historyPropagate*(
p: PortalProtocol, dataFile: string, verify = false
): Future[Result[void, string]] {.async.} =
const concurrentGossips = 20

var gossipQueue =
newAsyncQueue[(Opt[NodeId], ContentKeysList, seq[byte])](concurrentGossips)
var gossipWorkers: seq[Future[void]]

proc gossipWorker(p: PortalProtocol) {.async.} =
while true:
let (srcNodeId, keys, content) = await gossipQueue.popFirst()

discard await p.neighborhoodGossip(srcNodeId, keys, @[content])

for i in 0 ..< concurrentGossips:
gossipWorkers.add(gossipWorker(p))

let blockData = readJsonType(dataFile, BlockDataTable)
if blockData.isOk():
for b in blocks(blockData.get(), verify):
for i, value in b:
if i == 0:
# Note: Skipping propagation of headers here as they should be offered
# separately to be certain that bodies and receipts can be verified.
# TODO: Rename this chain of calls to be more clear about this and
# adjust the interator call.
continue
# Only sending non empty data, e.g. empty receipts are not send
# TODO: Could do a similar thing for a combination of empty
# txs and empty uncles, as then the serialization is always the same.
if value[1].len() > 0:
info "Seeding block content into the network", contentKey = value[0]
# Note: This is the slowest part due to the hashing that takes place.
let
encKey = history_content.encode(value[0])
contentId = history_content.toContentId(encKey)
p.storeContent(encKey, contentId, value[1])

await gossipQueue.addLast(
(Opt.none(NodeId), ContentKeysList(@[encode(value[0])]), value[1])
)

return ok()
else:
return err(blockData.error)

proc historyPropagateBlock*(
p: PortalProtocol, dataFile: string, blockHash: string, verify = false
): Future[Result[void, string]] {.async.} =
let blockDataTable = readJsonType(dataFile, BlockDataTable)

if blockDataTable.isOk():
let b =
try:
blockDataTable.get()[blockHash]
except KeyError:
return err("Block hash not found in block data file")

let blockDataRes = readBlockData(blockHash, b)
if blockDataRes.isErr:
return err(blockDataRes.error)

let blockData = blockDataRes.get()

for value in blockData:
info "Seeding block content into the network", contentKey = value[0]
let
encKey = history_content.encode(value[0])
contentId = history_content.toContentId(encKey)
p.storeContent(encKey, contentId, value[1])

discard await p.neighborhoodGossip(
Opt.none(NodeId), ContentKeysList(@[encode(value[0])]), @[value[1]]
)

return ok()
else:
return err(blockDataTable.error)

proc historyPropagateHeadersWithProof*(
p: PortalProtocol, epochHeadersFile: string, epochRecordFile: string
): Future[Result[void, string]] {.async.} =
let res = readBlockHeaders(epochHeadersFile)
if res.isErr():
return err(res.error)

let blockHeaders = res.get()

let epochRecordRes = readEpochRecordCached(epochRecordFile)
if epochRecordRes.isErr():
return err(res.error)

let epochRecord = epochRecordRes.get()
for header in blockHeaders:
if header.isPreMerge():
let headerWithProof = buildHeaderWithProof(header, epochRecord)
if headerWithProof.isErr:
return err(headerWithProof.error)

let
content = headerWithProof.get()
contentKey = ContentKey(
contentType: blockHeader,
blockHeaderKey: BlockKey(blockHash: header.rlpHash()),
)
encKey = history_content.encode(contentKey)
contentId = history_content.toContentId(encKey)
encodedContent = SSZ.encode(content)

p.storeContent(encKey, contentId, encodedContent)

let keys = ContentKeysList(@[encode(contentKey)])
discard await p.neighborhoodGossip(Opt.none(NodeId), keys, @[encodedContent])

return ok()

proc historyPropagateHeadersWithProof*(
p: PortalProtocol, dataDir: string
): Future[Result[void, string]] {.async.} =
for i in 0 ..< preMergeEpochs:
let
epochHeadersfile =
try:
dataDir / &"mainnet-headers-epoch-{i.uint64:05}.e2s"
except ValueError as e:
raiseAssert e.msg
epochRecordFile =
try:
dataDir / &"mainnet-epoch-record-{i.uint64:05}.ssz"
except ValueError as e:
raiseAssert e.msg

let res =
await p.historyPropagateHeadersWithProof(epochHeadersfile, epochRecordFile)
if res.isOk():
info "Finished gossiping 1 epoch of headers with proof", i
else:
return err(res.error)

return ok()

proc historyPropagateHeaders*(
p: PortalProtocol, dataFile: string, verify = false
): Future[Result[void, string]] {.async.} =
# TODO: Should perhaps be integrated with `historyPropagate` call.
const concurrentGossips = 20

var gossipQueue = newAsyncQueue[(ContentKeysList, seq[byte])](concurrentGossips)
var gossipWorkers: seq[Future[void]]

proc gossipWorker(p: PortalProtocol) {.async.} =
while true:
let (keys, content) = await gossipQueue.popFirst()

discard await p.neighborhoodGossip(Opt.none(NodeId), keys, @[content])

for i in 0 ..< concurrentGossips:
gossipWorkers.add(gossipWorker(p))

let blockData = readJsonType(dataFile, BlockDataTable)
if blockData.isOk():
for header in headers(blockData.get(), verify):
info "Seeding header content into the network", contentKey = header[0]
let
encKey = history_content.encode(header[0])
contentId = history_content.toContentId(encKey)
p.storeContent(encKey, contentId, header[1])

await gossipQueue.addLast((ContentKeysList(@[encode(header[0])]), header[1]))

return ok()
else:
return err(blockData.error)

##
## Era1 based iterators that encode to Portal content
##
Expand Down
4 changes: 0 additions & 4 deletions fluffy/rpc/rpc_calls/rpc_portal_debug_calls.nim
View file
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,3 @@ createRpcSigsFromNim(RpcClient):

proc portal_debug_historyGossipHeaders(era1File: string): bool
proc portal_debug_historyGossipBlockContent(era1File: string): bool
proc portal_debug_history_storeContent(dataFile: string): bool
proc portal_debug_history_propagate(dataFile: string): bool
proc portal_debug_history_propagateHeaders(dataFile: string): bool
proc portal_debug_history_propagateBlock(dataFile: string, blockHash: string): bool
Loading

0 comments on commit a5f0b12

Please sign in to comment.