Build tasks for HTTP retrievals only + various improvements (#7)

* feat: skip deals created before June 2023 Signed-off-by: Miroslav Bajtoš <[email protected]> * feat: query IPNI with concurrency=5 Signed-off-by: Miroslav Bajtoš <[email protected]> * cache IPNI responses Signed-off-by: Miroslav Bajtoš <[email protected]> * rename "stats.retrievable" to "stats.advertised" Signed-off-by: Miroslav Bajtoš <[email protected]> * log unknown protocol codes Signed-off-by: Miroslav Bajtoš <[email protected]> * keep HTTP retrievals only Signed-off-by: Miroslav Bajtoš <[email protected]> * fix loading of cached IPNI responses Signed-off-by: Miroslav Bajtoš <[email protected]> * log progress of build-retrieval-tasks Signed-off-by: Miroslav Bajtoš <[email protected]> * document how to disable MaxListenersExceededWarning Signed-off-by: Miroslav Bajtoš <[email protected]> * treat grapsync providers at `/tcp/80/http` as trustless HTTP GW Signed-off-by: Miroslav Bajtoš <[email protected]> * feat: process StateMarketDeals in Rust Speed up the initial JSON parsing from ~1h to <3m. Signed-off-by: Miroslav Bajtoš <[email protected]> * fix: add missing "const" Signed-off-by: Miroslav Bajtoš <[email protected]> * feat: dockerize + serve output via HTTP Signed-off-by: Miroslav Bajtoš <[email protected]> * npx standard --fix Signed-off-by: Miroslav Bajtoš <[email protected]> * update README Signed-off-by: Miroslav Bajtoš <[email protected]> --------- Signed-off-by: Miroslav Bajtoš <[email protected]>
CheckerNetwork · Dec 7, 2023 · 1e4b134 · 1e4b134
1 parent 26fbfef
commit 1e4b134
Show file tree

Hide file tree

Showing 10 changed files with 616 additions and 27 deletions.
diff --git a/.gitignore b/.gitignore
@@ -134,3 +134,8 @@ generated/ldn-deals.ndjson
 generated/retrieval-tasks.ndjson
 generated/StateMarketDeals.ndjson
 generated/update-spark-db.sql
+
+
+# Added by cargo
+
+/target
diff --git a/Cargo.lock b/Cargo.lock
diff --git a/Cargo.toml b/Cargo.toml
@@ -0,0 +1,11 @@
+[package]
+name = "fil-deal-ingester"
+version = "0.1.0"
+edition = "2021"
+
+# See more keys and their definitions at https://doc.rust-lang.org/cargo/reference/manifest.html
+
+[dependencies]
+env_logger = "0.10.1"
+json-event-parser = "0.1.1"
+log = "0.4.20"
diff --git a/Dockerfile b/Dockerfile
@@ -0,0 +1,11 @@
+FROM node:20
+USER node
+WORKDIR /usr/src/app
+COPY package*.json .
+COPY generated/ldn-deals.ndjson .
+COPY scripts scripts
+RUN ls -l
+RUN npm ci
+
+ENV SERVE=1
+CMD [ "node", "--no-warnings", "scripts/build-retrieval-tasks.js", "ldn-deals.ndjson" ]