Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Build tasks for HTTP retrievals only + various improvements #7

Merged
merged 15 commits into from
Dec 7, 2023
Merged

Build tasks for HTTP retrievals only + various improvements #7

merged 15 commits into from
Dec 7, 2023

Conversation

bajtos
Copy link
Member

@bajtos bajtos commented Dec 6, 2023 •
edited
Loading

  • skip deals created before June 2023
  • query IPNI with concurrency=5
  • cache IPNI responses
  • keep HTTP retrievals only
  • treat grapsync providers at /tcp/80/http as trustless HTTP GW
  • don't append p2p identity to multiaddr - it's not used by HTTP
  • dockerize + deploy to Fly.io
  • process StateMarketDeals.json in Rust to speed up the initial JSON parsing from ~1h to <3m

Links:

bajtos added 11 commits December 4, 2023 16:36
@bajtos
feat: skip deals created before June 2023
1fd6713 
Signed-off-by: Miroslav Bajtoš <[email protected]>
@bajtos
feat: query IPNI with concurrency=5
d55d7b3 
Signed-off-by: Miroslav Bajtoš <[email protected]>
@bajtos
cache IPNI responses
91c5ce3 
Signed-off-by: Miroslav Bajtoš <[email protected]>
@bajtos
rename "stats.retrievable" to "stats.advertised"
d4cc9ab 
Signed-off-by: Miroslav Bajtoš <[email protected]>
@bajtos
log unknown protocol codes
6ad2388 
Signed-off-by: Miroslav Bajtoš <[email protected]>
@bajtos
keep HTTP retrievals only
669925d 
Signed-off-by: Miroslav Bajtoš <[email protected]>
@bajtos
fix loading of cached IPNI responses
19b8bfe 
Signed-off-by: Miroslav Bajtoš <[email protected]>
@bajtos
log progress of build-retrieval-tasks
649e167 
Signed-off-by: Miroslav Bajtoš <[email protected]>
@bajtos
document how to disable MaxListenersExceededWarning
e72e914 
Signed-off-by: Miroslav Bajtoš <[email protected]>
@bajtos
treat grapsync providers at /tcp/80/http as trustless HTTP GW
42ab058 
Signed-off-by: Miroslav Bajtoš <[email protected]>
@bajtos
feat: process StateMarketDeals in Rust
105859e 
Speed up the initial JSON parsing from ~1h to <3m.

Signed-off-by: Miroslav Bajtoš <[email protected]>
bajtos added 3 commits December 7, 2023 10:32
@bajtos
fix: add missing "const"
ff8b9d2 
Signed-off-by: Miroslav Bajtoš <[email protected]>
@bajtos
feat: dockerize + serve output via HTTP
26914f1 
Signed-off-by: Miroslav Bajtoš <[email protected]>
@bajtos
npx standard --fix
a482940 
Signed-off-by: Miroslav Bajtoš <[email protected]>
@bajtos bajtos marked this pull request as ready for review December 7, 2023 10:24
@bajtos bajtos requested a review from juliangruber December 7, 2023 10:24
@bajtos bajtos mentioned this pull request Dec 7, 2023
Merged
@bajtos
update README
7e75b5a 
Signed-off-by: Miroslav Bajtoš <[email protected]>
@bajtos bajtos changed the title Small improvements for the second ingestion run Improvements for the second ingestion run Dec 7, 2023
@bajtos bajtos changed the title Improvements for the second ingestion run Build tasks for HTTP retrievals only + various improvements Dec 7, 2023
juliangruber
juliangruber approved these changes Dec 7, 2023
View reviewed changes
Copy link
Member

@juliangruber juliangruber left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not seeing anything sticking out, but also I'm not yet familiar with this codebase

@bajtos bajtos merged commit 1e4b134 into main Dec 7, 2023
@bajtos bajtos deleted the another-round branch December 7, 2023 14:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@juliangruber juliangruber juliangruber approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@bajtos @juliangruber