feat: process StateMarketDeals in Rust

Speed up the initial JSON parsing from ~1h to <3m.

Signed-off-by: Miroslav Bajtoš <[email protected]>

.gitignore

@@ -134,3 +134,8 @@ generated/ldn-deals.ndjson
 generated/retrieval-tasks.ndjson
 generated/StateMarketDeals.ndjson
 generated/update-spark-db.sql
+
+
+# Added by cargo
+
+/target

Cargo.toml

@@ -0,0 +1,11 @@
+[package]
+name = "fil-deal-ingester"
+version = "0.1.0"
+edition = "2021"
+
+# See more keys and their definitions at https://doc.rust-lang.org/cargo/reference/manifest.html
+
+[dependencies]
+env_logger = "0.10.1"
+json-event-parser = "0.1.1"
+log = "0.4.20"

README.md

@@ -30,10 +30,16 @@ The output is committed to git, see [./generated/ldn-clients.csv](./generated/ld
 
    WARNING: The decompressed file has over 23 GB
 
-3. Run
+3. Build the tool for converting `StateMarketDeals.json` to newline-delimited JSON
+
+   ```sh
+   cargo build --release
+   ```
+
+4. Run
 
    ```sh
-   jq --stream -c 'fromstream(1|truncate_stream(inputs))' StateMarketDeals.json > generated/StateMarketDeals.ndjson
+   ./target/release/fil-deal-ingester StateMarketDeals.json > generated/StateMarketDeals.ndjson
    ```
 
    WARNING: This will take very long (more than 1 hour).