Code

GRPC Ingest#183

Code

Draft

kespinola wants to merge 77 commits intometaplex-foundation:mainmetaplex-foundation/digital-asset-rpc-infrastructure:mainfrom rpcpool:grpc-ingestrpcpool/digital-asset-rpc-infrastructure:grpc-ingestCopy head branch name to clipboard

+7,404-2,264

Commits

Commits on Apr 16, 2024

• fmt

  

  fanatid

  authored and

  kespinola

  committed
  ea5379e

  Copy full SHA for ea5379e

• clippy

  

  fanatid

  authored and

  kespinola

  committed
  8b12316

  Copy full SHA for 8b12316

• ci: add lock, fmt, clippy checks

  

  fanatid

  authored and

  kespinola

  committed
  974c90f

  Copy full SHA for 974c90f

• include migration to workspace

  

  fanatid

  authored and

  kespinola

  committed
  788973e

  Copy full SHA for 788973e

• include das_api to workspace

  

  fanatid

  authored and

  kespinola

  committed
  7a870c2

  Copy full SHA for 7a870c2

• add global clippy lints

  

  fanatid

  authored and

  kespinola

  committed
  7e3f9ef

  Copy full SHA for 7e3f9ef

• use workspace

  

  fanatid

  authored and

  kespinola

  committed
  aa916a4

  Copy full SHA for aa916a4

• add crate program_transformers

  

  fanatid

  authored and

  kespinola

  committed
  033873d

  Copy full SHA for 033873d

• nft_ingester: use program_transformers crate

  

  fanatid

  authored and

  kespinola

  committed
  1008f86

  Copy full SHA for 1008f86

• remove not used deps

  

  fanatid

  authored and

  kespinola

  committed
  83b171e

  Copy full SHA for 83b171e

• remove AccountInfo

  

  fanatid

  authored and

  kespinola

  committed
  9d8031e

  Copy full SHA for 9d8031e

• remove plerkle from program_transformers

  

  fanatid

  authored and

  kespinola

  committed
  096616d

  Copy full SHA for 096616d

• nft_ingester2: grpc2redis

  

  fanatid

  authored and

  kespinola

  committed
  ef0b0e9

  Copy full SHA for ef0b0e9

• add redis streaming for ingester

  

  fanatid

  authored and

  kespinola

  committed
  076bffa

  Copy full SHA for 076bffa

• create pg pool

  

  fanatid

  authored and

  kespinola

  committed
  d8673cf

  Copy full SHA for d8673cf

• parse incoming message from redis

  

  fanatid

  authored and

  kespinola

  committed
  c93adff

  Copy full SHA for c93adff

• add force shutdown with signals

  

  fanatid

  authored and

  kespinola

  committed
  53fa481

  Copy full SHA for 53fa481

• insert download metadata tasks

  

  fanatid

  authored and

  kespinola

  committed
  11b4151

  Copy full SHA for 11b4151

• download-metadata subtask

  

  fanatid

  authored and

  kespinola

  committed
  8fca2c2

  Copy full SHA for 8fca2c2

• refactor: rename nft_ingest2

  kespinola

  committed
  fcc18d6

  Copy full SHA for fcc18d6

• fix: applying account and transction filters to grpc subscription request

  kespinola

  committed
  294fa1a

  Copy full SHA for 294fa1a

Commits on May 2, 2024

• refactor: take out multiple connections to dragonmouth and then have the main thread keep a chache of seen events so that it doesnt push to ingest if seen by another connections

  kespinola

  committed
  98a7e4e

  Copy full SHA for 98a7e4e

Commits on May 3, 2024

• refactor: switch to accepting multiple dragonmouth endoints to mimic dual validators writing to ingest stream

  kespinola

  committed
  997d515

  Copy full SHA for 997d515

Commits on May 9, 2024

• Merge branch 'main' of github.com:rpcpool/digital-asset-rpc-infrastructure into grpc-ingest

  kespinola

  committed
  75c4115

  Copy full SHA for 75c4115

• fix: clippy errors

  kespinola

  committed
  67cc21e

  Copy full SHA for 67cc21e

Commits on Jul 17, 2024

• fix: processes transactions for a tree sequentially to make sure there is on out of order processing. process the transaction by ops instead of queueing to TXN_FILL redis stream.

  kespinola

  committed
  5f08298

  Copy full SHA for 5f08298

• refactor: use program transform in the account backfill no queuing into redis.

  kespinola

  committed
  0eced5d

  Copy full SHA for 0eced5d

Commits on Jul 18, 2024

• refactor: move bubble backfill to a lib so can be used by other projects eg LightDAS and be included in the grpc-ingest to execute backfills on restart then pick up tasks from the stream to insure …

  Show description for 138bf2e
  kespinola

  committed
  138bf2e

  Copy full SHA for 138bf2e

Commits on Jul 19, 2024

• Merge remote-tracking branch 'origin' into grpc-ingest

  kespinola

  committed
  d80e7b8

  Copy full SHA for d80e7b8

Commits on Jul 30, 2024

• chor: refactor grpc command to use topograph to clean up the control flow. also pull in backfill to be used in ingest

  kespinola

  committed
  2cc3411

  Copy full SHA for 2cc3411

Commits on Aug 9, 2024

• chore: switch to run_v2 for ingest which handles ingesting accounts and transactions

  kespinola

  committed
  81f5d71

  Copy full SHA for 81f5d71

Commits on Aug 10, 2024

• Merge remote-tracking branch 'origin' into grpc-ingest

  kespinola

  committed
  10c46d5

  Copy full SHA for 10c46d5

Commits on Aug 12, 2024

• chore: handle metadata json within the ingest command

  kespinola

  committed
  a151630

  Copy full SHA for a151630

• test: metadata json stream processing

  kespinola

  committed
  f398880

  Copy full SHA for f398880

Commits on Aug 14, 2024

• feat: set num threads for topograph. report on postgres and redis in ingester using dedicate exec

  kespinola

  committed
  70eef04

  Copy full SHA for 70eef04

Commits on Aug 20, 2024

• chore: switch to topograph for thread management. create separate ingest stream which handle reading messages and handling messages.

  kespinola

  committed
  77a98f6

  Copy full SHA for 77a98f6

Commits on Aug 22, 2024

• fix: config and ack

  kespinola

  committed
  0db0612

  Copy full SHA for 0db0612

• hack: disable pending processing while see why messages arent being deleted

  kespinola

  committed
  32f9852

  Copy full SHA for 32f9852

• fix: exit early if slot matches metadata json download and reindex is false.

  kespinola

  committed
  bab4678

  Copy full SHA for bab4678

• fix: picking dangling pending messages before reading new

  kespinola

  committed
  0c4f872

  Copy full SHA for 0c4f872

Commits on Aug 23, 2024

• fix: ingest stream shutdown and requesting snapshots

  kespinola

  committed
  eb75ca3

  Copy full SHA for eb75ca3

• fix: processing pending. add metrics for total workers.

  kespinola

  committed
  3213080

  Copy full SHA for 3213080

Commits on Aug 26, 2024

• fix: bubble backfill transform. no op out of download metadata info.

  kespinola

  committed
  7f3b391

  Copy full SHA for 7f3b391

Commits on Sep 13, 2024

• Add force flag to TreeWorkerArgs for complete reindexing (#148)

  Show description for 4c3f649
  

  kevinrodriguez-io

  and

  kespinola

  authored
  4c3f649

  Copy full SHA for 4c3f649

Commits on Sep 16, 2024

• refactor: drop exculsion clauses on cl_items to allow for reindexing (#151)

  kespinola

  authored
  4e3c076

  Copy full SHA for 4e3c076

Commits on Sep 26, 2024

• feat: replay bubblegum transactions based on what is in cl_audits_v2

  kespinola

  committed
  c789b52

  Copy full SHA for c789b52

Commits on Oct 2, 2024

• fix: order bubblegum backfill instructions by seq (#158)

  kespinola

  authored
  45a5dcf

  Copy full SHA for 45a5dcf

Commits on Oct 7, 2024

• include back seq checks

  kespinola

  committed
  f4744c0

  Copy full SHA for f4744c0

• Remove enforcing seq in bubblegum backfill

  kespinola

  committed
  1afee45

  Copy full SHA for 1afee45

Commits on Oct 8, 2024

• Replay single tree with targeted seq replay (#159)

  kespinola

  authored
  18411a9

  Copy full SHA for 18411a9

Commits on Oct 9, 2024

• Support Account Snapshot Flushes (#154)

  Show description for 110f71c
  

  kespinola

  and

  kevinrodriguez-io

  authored
  110f71c

  Copy full SHA for 110f71c

• fix: keep grpc alive by sending using ping-pong (#160)

  Nagaprasadvr

  authored
  118baec

  Copy full SHA for 118baec

• Only Time Based Flushes (#161)

  Show description for 01bbe99
  kespinola

  authored
  01bbe99

  Copy full SHA for 01bbe99

Commits on Oct 11, 2024

• delete and recreate the consumer (#162)

  kespinola

  authored
  cde4594

  Copy full SHA for cde4594

Commits on Oct 12, 2024

• Type out redis message handlers (#163)

  kespinola

  authored
  0005b3e

  Copy full SHA for 0005b3e

Commits on Oct 13, 2024

• skip reading new messages if the ack buffer is full. track running tasks. (#164)

  kespinola

  authored
  d092ad8

  Copy full SHA for d092ad8

Commits on Oct 14, 2024

• only push handler job for ingest if capacity (#165)

  kespinola

  authored
  6f158f3

  Copy full SHA for 6f158f3

• Only throttle based on the task semaphore (#166)

  kespinola

  authored
  a347066

  Copy full SHA for a347066

• Remove topograph from ingest command (#167)

  kespinola

  authored
  188c878

  Copy full SHA for 188c878

Commits on Oct 21, 2024

• Configure database connection idle timeout and max lifetime (#169)

  kespinola

  authored
  e15172f

  Copy full SHA for e15172f

Commits on Oct 22, 2024

• Add single NFT fetch and backfill feature to ops (#170)

  Show description for fdbf03a
  Nagaprasadvr

  authored
  fdbf03a

  Copy full SHA for fdbf03a

Commits on Oct 23, 2024

• Add lock timeout on v1_asset statements

  kespinola

  committed
  d7cc058

  Copy full SHA for d7cc058

• Set the lock timeout to 5s

  kespinola

  committed
  17d7e0a

  Copy full SHA for 17d7e0a

Commits on Oct 29, 2024

• Bubblegum verify (#150)

  Show description for 99b59c0
  kespinola

  authored
  99b59c0

  Copy full SHA for 99b59c0

Commits on Nov 8, 2024

• Separate global stream to multiple Program separated streams and handle each stream (#168)

  Show description for d3bfe1b
  

  Nagaprasadvr

  and

  kespinola

  authored
  d3bfe1b

  Copy full SHA for d3bfe1b

• Report proofs as they come (#178)

  kespinola

  authored
  f311e14

  Copy full SHA for f311e14

Commits on Nov 11, 2024

• Buffer redis messages (#179)

  kespinola

  authored
  f8bdef2

  Copy full SHA for f8bdef2

• Fix tracking ack tasks

  kespinola

  committed
  776ae0a

  Copy full SHA for 776ae0a

• fix: add back ack tasks total

  kespinola

  committed
  151168e

  Copy full SHA for 151168e

Commits on Nov 12, 2024

• recreate consumer but not groups (#181)

  kespinola

  authored
  ae11907

  Copy full SHA for ae11907

Commits on Nov 14, 2024

• Time Ingest jobs (#182)

  Show description for 80d61dc
  kespinola

  authored
  80d61dc

  Copy full SHA for 80d61dc

Commits on Nov 27, 2024

• Use Example Configs (#180)

  Show description for 4baaca6
  

  kespinola

  and

  Ahzam Akhtar

  authored
  4baaca6

  Copy full SHA for 4baaca6

Commits on Dec 18, 2024

• Respect commitment level set in grpc config (#191)

  kespinola

  authored
  c235125

  Copy full SHA for c235125

Commits on Jan 15, 2025

• Merge remote-tracking branch 'origin/main' into grpc-ingest

  kespinola

  committed
  a464fd3

  Copy full SHA for a464fd3

Commits on Jan 20, 2025

• Use tokio join set ingest stream task tracking

  kespinola

  committed
  9f3d543

  Copy full SHA for 9f3d543

Commits on Jan 21, 2025

• Add metrics to track metadata_json_download success and failure (#207)

  Show description for 3d7a88d
  

  Nagaprasadvr

  and

  kespinola

  authored
  3d7a88d

  Copy full SHA for 3d7a88d

• add retry config to download_metadata_json task (#208)

  Nagaprasadvr

  authored
  44ce53f

  Copy full SHA for 44ce53f