-
Notifications
You must be signed in to change notification settings - Fork 41
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
GRPC Ingest #183
Draft
kespinola
wants to merge
77
commits into
metaplex-foundation:main
Choose a base branch
from
rpcpool:grpc-ingest
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
+7,404
−2,264
Draft
GRPC Ingest #183
Changes from all commits
Commits
Show all changes
77 commits
Select commit
Hold shift + click to select a range
ea5379e
fmt
fanatid 8b12316
clippy
fanatid 974c90f
ci: add lock, fmt, clippy checks
fanatid 788973e
include migration to workspace
fanatid 7a870c2
include das_api to workspace
fanatid 7e3f9ef
add global clippy lints
fanatid aa916a4
use workspace
fanatid 033873d
add crate program_transformers
fanatid 1008f86
nft_ingester: use program_transformers crate
fanatid 83b171e
remove not used deps
fanatid 9d8031e
remove AccountInfo
fanatid 096616d
remove plerkle from program_transformers
fanatid ef0b0e9
nft_ingester2: grpc2redis
fanatid 076bffa
add redis streaming for ingester
fanatid d8673cf
create pg pool
fanatid c93adff
parse incoming message from redis
fanatid 53fa481
add force shutdown with signals
fanatid 11b4151
insert download metadata tasks
fanatid 8fca2c2
download-metadata subtask
fanatid fcc18d6
refactor: rename nft_ingest2
kespinola 294fa1a
fix: applying account and transction filters to grpc subscription req…
kespinola 98a7e4e
refactor: take out multiple connections to dragonmouth and then have …
kespinola 997d515
refactor: switch to accepting multiple dragonmouth endoints to mimic …
kespinola 75c4115
Merge branch 'main' of github.com:rpcpool/digital-asset-rpc-infrastru…
kespinola 67cc21e
fix: clippy errors
kespinola 5f08298
fix: processes transactions for a tree sequentially to make sure ther…
kespinola 0eced5d
refactor: use program transform in the account backfill no queuing in…
kespinola 138bf2e
refactor: move bubble backfill to a lib so can be used by other proje…
kespinola d80e7b8
Merge remote-tracking branch 'origin' into grpc-ingest
kespinola 2cc3411
chor: refactor grpc command to use topograph to clean up the control …
kespinola 81f5d71
chore: switch to run_v2 for ingest which handles ingesting accounts a…
kespinola 10c46d5
Merge remote-tracking branch 'origin' into grpc-ingest
kespinola a151630
chore: handle metadata json within the ingest command
kespinola f398880
test: metadata json stream processing
kespinola 70eef04
feat: set num threads for topograph. report on postgres and redis in …
kespinola 77a98f6
chore: switch to topograph for thread management. create separate ing…
kespinola 0db0612
fix: config and ack
kespinola 32f9852
hack: disable pending processing while see why messages arent being d…
kespinola bab4678
fix: exit early if slot matches metadata json download and reindex is…
kespinola 0c4f872
fix: picking dangling pending messages before reading new
kespinola eb75ca3
fix: ingest stream shutdown and requesting snapshots
kespinola 3213080
fix: processing pending. add metrics for total workers.
kespinola 7f3b391
fix: bubble backfill transform. no op out of download metadata info.
kespinola 4c3f649
Add force flag to TreeWorkerArgs for complete reindexing (#148)
kevinrodriguez-io 4e3c076
refactor: drop exculsion clauses on cl_items to allow for reindexing …
kespinola c789b52
feat: replay bubblegum transactions based on what is in cl_audits_v2
kespinola 45a5dcf
fix: order bubblegum backfill instructions by seq (#158)
kespinola f4744c0
include back seq checks
kespinola 1afee45
Remove enforcing seq in bubblegum backfill
kespinola 18411a9
Replay single tree with targeted seq replay (#159)
kespinola 110f71c
Support Account Snapshot Flushes (#154)
kespinola 118baec
fix: keep grpc alive by sending using ping-pong (#160)
Nagaprasadvr 01bbe99
Only Time Based Flushes (#161)
kespinola cde4594
delete and recreate the consumer (#162)
kespinola 0005b3e
Type out redis message handlers (#163)
kespinola d092ad8
skip reading new messages if the ack buffer is full. track running ta…
kespinola 6f158f3
only push handler job for ingest if capacity (#165)
kespinola a347066
Only throttle based on the task semaphore (#166)
kespinola 188c878
Remove topograph from ingest command (#167)
kespinola e15172f
Configure database connection idle timeout and max lifetime (#169)
kespinola fdbf03a
Add single NFT fetch and backfill feature to ops (#170)
Nagaprasadvr d7cc058
Add lock timeout on v1_asset statements
kespinola 17d7e0a
Set the lock timeout to 5s
kespinola 99b59c0
Bubblegum verify (#150)
kespinola d3bfe1b
Separate global stream to multiple Program separated streams and hand…
Nagaprasadvr f311e14
Report proofs as they come (#178)
kespinola f8bdef2
Buffer redis messages (#179)
kespinola 776ae0a
Fix tracking ack tasks
kespinola 151168e
fix: add back ack tasks total
kespinola ae11907
recreate consumer but not groups (#181)
kespinola 80d61dc
Time Ingest jobs (#182)
kespinola 4baaca6
Use Example Configs (#180)
kespinola c235125
Respect commitment level set in grpc config (#191)
kespinola a464fd3
Merge remote-tracking branch 'origin/main' into grpc-ingest
kespinola 9f3d543
Use tokio join set ingest stream task tracking
kespinola 3d7a88d
Add metrics to track metadata_json_download success and failure (#207)
Nagaprasadvr 44ce53f
add retry config to download_metadata_json task (#208)
Nagaprasadvr File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
yellowstone crates are referenced using git so require network fetch to retrieve which violates frozen flag.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
https://lib.rs/crates/yellowstone-grpc-client
NVM crate is published