feat: add `S3Stage` downloader #13784

joshieDo · 2025-01-13T11:44:36Z

adds S3Stage skeleton logic. It's incomplete and not enabled. TODOs can be disregarded for now.
adds S3Stage downloader. Downloads a file in parallel allowing resumes after shutdowns.

/// Downloads file from url to data file path.
///
/// If a `file_hash` is passed, it will verify it at the end.
///
/// ## Details
///
/// 1) A [`Metadata`] file is created or opened in `{target_dir}/download/{filename}.metadata`. It
///    tracks the download progress including total file size, downloaded bytes, chunk sizes, and
///    ranges that still need downloading. Allows for resumability.
/// 2) The target file is preallocated with the total size of the file in
///    `{target_dir}/download/{filename}`.
/// 3) Multiple `workers` are spawned for downloading of specific chunks of the file.
/// 4) `Orchestrator` manages workers, distributes chunk ranges, and ensures the download progresses
///    efficiently by dynamically assigning tasks to workers as they become available.
/// 5) Once the file is downloaded:
///     * If `file_hash` is `Some`, verifies its blake3 hash.
///     * Deletes the metadata file
///     * Moves downloaded file to target directory.

mattsse

cool,

I mostly have some questions re async or rather how we integrate the downloader in the s3stage

crates/stages/stages/src/stages/s3/downloader/error.rs

crates/stages/stages/src/stages/s3/downloader/fetch.rs

crates/stages/stages/src/stages/s3/downloader/meta.rs

crates/stages/stages/src/stages/s3/downloader/worker.rs

crates/stages/stages/src/stages/s3/mod.rs

mattsse

have a few more questions about workers <> orchestrators

mattsse · 2025-01-15T13:45:22Z

crates/stages/stages/src/stages/s3/downloader/fetch.rs

+    let client = Client::new();
+    let resp = client.head(url).send().await?;
+    let total_length: usize = resp
+        .headers()
+        .get(CONTENT_LENGTH)
+        .and_then(|v| v.to_str().ok())
+        .and_then(|s| s.parse().ok())


do we need any additional header setup?

no, that's it. just need to know the total size of the file

crates/stages/stages/src/stages/s3/downloader/meta.rs

crates/stages/stages/src/stages/s3/mod.rs

mattsse · 2025-01-15T13:51:54Z

crates/stages/stages/src/stages/s3/mod.rs

+        // Spawns the downloader task
+        self.spawn_fetch(input);


this will always be called when we call the poll function? this seems wrong, because we don't track whether we've downloaded everything

no, just when self.fetch_rx is None, meaning that there's no task in the background running. I should make it more explicit.

now that i look at it, there's an issue inside spawn_fetch that will always cause it to spawn, when it should only when theres files to be downloaded (maybe that's what you mean?)

mattsse · 2025-01-15T13:53:49Z

crates/stages/stages/src/stages/s3/downloader/fetch.rs

+        // Distribute chunk ranges to workers when they free up
+        while let Some(worker_msg) = orchestrator_rx.recv().await {


is this more performant than using a single downloader? I'd assume downloading is bound by bandwidth so unclear if using multiple downloaders improves this?

Although in theory it's bound by bandwidth, many times servers will throttle single connections. Using multiple connections helps

crates/stages/stages/src/stages/s3/downloader/worker.rs

crates/stages/stages/src/stages/s3/downloader/fetch.rs

mattsse · 2025-01-21T12:14:22Z

crates/stages/stages/src/stages/s3/downloader/fetch.rs

+/// Ensures the file on path has the expected blake3 hash.
+fn check_file_hash(path: &Path, expected: &B256) -> Result<(), DownloaderError> {
+    let mut reader = BufReader::new(File::open(path)?);
+    let mut hasher = blake3::Hasher::new();


do we need blake3 here or can use sha256?

blake3 is way faster, and we're dealing with total 300GB

crates/stages/stages/src/stages/s3/downloader/worker.rs

mattsse · 2025-01-21T12:19:29Z

crates/stages/stages/src/stages/s3/mod.rs

+        // Re-initializes the provider to detect the new additions
+        provider.static_file_provider().initialize_index()?;
+
+        // TODO logic for appending tx_block


is this still a todo or is init index all we need?

should this reinit per invocation or only once after we previously downloaded the file?

still TODO, goal of the PR is just the downloader. follow-up would implement the stage logic itself

crates/stages/stages/src/stages/s3/mod.rs

mattsse

good start, let's track remaining todos and submit as followups?

joshieDo added C-enhancement New feature or request A-staged-sync Related to staged sync (pipelines and stages) A-downloaders Related to headers/bodies downloaders labels Jan 13, 2025

joshieDo requested a review from mattsse January 13, 2025 11:44

joshieDo requested review from onbjerg, rkrasiuk, shekhirin and gakonst as code owners January 13, 2025 11:44

joshieDo requested a review from Rjected January 13, 2025 11:45

joshieDo force-pushed the joshie/s3stage branch 8 times, most recently from 79066e8 to 43e6188 Compare January 13, 2025 13:28

mattsse reviewed Jan 13, 2025

View reviewed changes

add downloader

e7cdd92

joshieDo force-pushed the joshie/s3stage branch 2 times, most recently from 8d67276 to 7efb229 Compare January 14, 2025 16:53

async

61811df

joshieDo force-pushed the joshie/s3stage branch from 7efb229 to 61811df Compare January 14, 2025 17:04

mattsse requested changes Jan 15, 2025

View reviewed changes

joshieDo force-pushed the joshie/s3stage branch 3 times, most recently from 44cc6e6 to c910229 Compare January 15, 2025 23:56

maybe_spawn_fetch

e59f489

joshieDo force-pushed the joshie/s3stage branch 2 times, most recently from 6c631be to 6ae894e Compare January 16, 2025 11:32

joshieDo force-pushed the joshie/s3stage branch from 6ae894e to a591e83 Compare January 20, 2025 23:53

MetadataFile & RemainingChunkRange

3331f34

joshieDo force-pushed the joshie/s3stage branch from a591e83 to 3331f34 Compare January 20, 2025 23:56

joshieDo requested a review from mattsse January 21, 2025 00:25

mattsse requested changes Jan 21, 2025

View reviewed changes

review

1c56571

joshieDo force-pushed the joshie/s3stage branch from 5a21474 to 1c56571 Compare January 21, 2025 13:55

mattsse approved these changes Jan 21, 2025

View reviewed changes

joshieDo added this pull request to the merge queue Jan 21, 2025

Merged via the queue into main with commit 6c3b1b8 Jan 21, 2025
43 checks passed

joshieDo deleted the joshie/s3stage branch January 21, 2025 18:22

refcell pushed a commit to ethereum-optimism/op-reth that referenced this pull request Jan 24, 2025

feat: add S3Stage downloader (paradigmxyz#13784)

a8dc212

This was referenced Jan 27, 2025

Add BlockBodies & BlockBodyIndices to StaticFiles #13264

Open

Add new stage for downloading pre-merge data #13468

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: add `S3Stage` downloader #13784

feat: add `S3Stage` downloader #13784

joshieDo commented Jan 13, 2025 •

edited

Loading

mattsse left a comment

mattsse left a comment

mattsse Jan 15, 2025

joshieDo Jan 15, 2025

mattsse Jan 15, 2025

joshieDo Jan 15, 2025 •

edited

Loading

mattsse Jan 15, 2025

joshieDo Jan 15, 2025 •

edited

Loading

mattsse Jan 21, 2025

joshieDo Jan 21, 2025

mattsse Jan 21, 2025

joshieDo Jan 21, 2025

mattsse left a comment

		// Distribute chunk ranges to workers when they free up
		while let Some(worker_msg) = orchestrator_rx.recv().await {

feat: add S3Stage downloader #13784

feat: add S3Stage downloader #13784

Conversation

joshieDo commented Jan 13, 2025 • edited Loading

mattsse left a comment

Choose a reason for hiding this comment

mattsse left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

joshieDo Jan 15, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

joshieDo Jan 15, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mattsse left a comment

Choose a reason for hiding this comment

feat: add `S3Stage` downloader #13784

feat: add `S3Stage` downloader #13784

joshieDo commented Jan 13, 2025 •

edited

Loading

joshieDo Jan 15, 2025 •

edited

Loading

joshieDo Jan 15, 2025 •

edited

Loading