v0.10.1

Changelog

v0.10.1

• Fix panic in merger when no prior merged blocks exist

v0.10.0 (deleted)

Fixed

• Fixed an issue where the merger would get stuck when too many (more than 2000) one-block-files were lying around, with block numbers below the current bundle high boundary.
• Fixed tools check merged-blocks block hole detection behavior on missing ranges (bumped sf-tools)
• Fixed a deadlock issue related to s3 storage error handling (bumped dstore)
• Fixed Firehose gRPC listening address over plain text.
• Fixed automatic merging of files within the mindreader is much more robust then before.

Changed

• Renamed common atm 4 flags to blocks-cache:
  --common-blocks-cache-{enabled|dir|max-recent-entry-bytes|max-entry-by-age-bytes}
• The default text encoder use to encode log entries now emits the level when coloring is disabled.
• Default value for flag --mindreader-node-enforce-peers is now "", this has been changed because the default value was useful only in development when running a local node-manager as either the miner or a peering node.
• Updated all Firehose stack direct dependencies.
• Updated confusing flag behavior for --common-system-shutdown-signal-delay and its interaction with gRPC connection draining in firehose component sometimes preventing it from shutting down.
• Reporting an error is if flag merge-threshold-block-age is way too low (< 30s).

Added

• Added tools download-from-firehose command to fetch blocks and save them as merged-blocks files locally.
• Added cloud-gcp:// auth module (bumped dauth)
• Added block data file caching (called ATM), this is to reduce the memory usage of component keeping block objects in memory.
• Added transforms: LogFilter, MultiLogFilter, CallToFilter, MultiCallToFilter to only return transaction traces that match logs or called addresses.
• Added support for irreversibility indexes in firehose to prevent replaying reorgs when streaming old blocks.
• Added support for log and call indexes to skip old blocks that do not match any transform filter.

Removed

• Removed some old components that are not required by Firehose stack directly, the repository is as lean as it ca now.

v0.10.0-rc.5

Changed

• Renamed common atm 4 flags to blocks-cache:
  --common-blocks-cache-{enabled|dir|max-recent-entry-bytes|max-entry-by-age-bytes}

Fixed

• Fixed tools check merged-blocks block hole detection behavior on missing ranges (bumped sf-tools)
• Fixed a deadlock issue related to s3 storage error handling (bumped dstore)

Added

• Added tools download-from-firehose command to fetch blocks and save them as merged-blocks files locally.
• Added cloud-gcp:// auth module (bumped dauth)

v0.10.0-rc.4

Changelog

• 2b9183e Changed default flag value for --mindreader-node-enforce-peers to ""

v0.10.0-rc.1

Added

• Added block data file caching (called ATM), this is to reduce the memory usage of component keeping block objects in memory.
• Added transforms: LogFilter, MultiLogFilter, CallToFilter, MultiCallToFilter to only return transaction traces that match logs or called addresses.
• Added support for irreversibility indexes in firehose to prevent replaying reorgs when streaming old blocks.
• Added support for log and call indexes to skip old blocks that do not match any transform filter.

Changed

• Updated all Firehose stack direct dependencies.
• Updated confusing flag behavior for --common-system-shutdown-signal-delay and its interaction with gRPC connection draining in firehose component sometimes preventing it from shutting down.
• Reporting an error is if flag merge-threshold-block-age is way too low (< 30s).

Removed

• Removed some old components that are not required by Firehose stack directly, the repository is as lean as it ca now.

Fixed

• Fixed Firehose gRPC listening address over plain text.
• Fixed automatic merging of files within the mindreader is much more robust then before.

Full Changelog

• 3ac8ec4 Add NewEthBlockIndexer() for convenience
• 6273adf Add TestEthBlockIndexProvider_Matches()
• ad4e41d Add TestEthBlockIndexProvider_NextMatching()
• 44eb852 Add TestEthBlockIndexProvider_WithinRange()
• d21f4ac Add docstrings to EthBlockIndexer
• 85238a3 Add multi-log filter transform
• 57c299a Add stubs for BlockIndexProvider
• d341b4a Added changelog for v0.10.0-rc.1
• ad161f7 Added git push prior release to ensure GitHub created release creates the correct tag
• d1f1751 Bump node-manager (moving some backup related stuff in there)
• 633a782 Extract WriteIndex() and ReadIndex() to bstream
• 52cd6af Extract chain-agnostic ProcessBlock() logic to bstream
• dbb5101 Fixed release script when doing a real non dry run
• 58042c9 Implement Index filter on eth.Address and eth.Hash
• 2ccd1ae Instantiating ProtocolSpefificModules
• 6954b53 Merge pull request #24 from streamingfast/stepd/multi-filter
• c29b0e7 Merge pull request #25 from streamingfast/froch/block-indexer
• d96cf3a Merge remote-tracking branch 'origin/develop' into froch/block-indexer
• 52871e4 Remove LogAddressIndex implementations in favor of BlockIndex
• 4a299ae Removed blockmeta and trxdb from sf-ethereum public
• 6a2cc3a Removing snapshotter dependency
• 02e63d7 Replace NewLogAddressIndexProvider() with NewEthBlockIndexProvider()
• e42aeb5 Replace NewLogAddressIndexer() with NewEthBlockIndexer()
• 166858e Rework EthBlockIndexProvider to accept a filterFunc
• 6c3fe6a Sanitize go.mod a bit
• 7cfb743 Updated bunch of dependencies to get proper logger name
• 0f2b0f4 Updated dlauncher to latest version
• 31e4f56 Updated to correct released version
• af40cee Updated to latest dlauncher
• ff0fc01 Updated to latest dlauncher and logging library
• e528270 Updated to latest version of bstream, pbgo, dgrpc, relayer, merger and blockmeta
• c4b08b0 Updated to latest version of everything
• da46fb0 Updated to streamingfast/overseer
• 0583610 Wrap a bstream.transform.BlockIndexer as an EthBlockIndexer
• 34d9054 account indexing tool to also produce irreversible blocks and start from where it left off
• bfd49b2 add proper logging of log/call filter transforms
• fc70c15 adjust accoutn_indexer to start from lowest of irreversible/account index
• 21aca49 bump bstream for startblock index matching fix
• f73cd60 bump bstream so firehose always sends start-block even when filtered
• 321e255 bump bstream to fix firehose cursor on indexed filesource
• 7b4bbbf bump bstream with my refactors on froch PR
• add26a0 bump bstream, firehose to fix stop-block bug in indexes
• 738745b bump deps to fix mindreader merging directly
• c136aa6 bump node-manager to fix double-compressed oneblockfiles
• 30bde30 bump node-manager to fix file deletion
• 75649ed bump node-manager/merger to fix direct merging
• 878a640 bump node-manager: merge immediately on boundary
• 29a8fbb change 'transforms' to 'transform' in proto, bump some index libs
• 24ac3a1 fix confusing flag behavior for grpcShutdownGracePeriod sometimes preventing firehose from shutting down
• 50787c9 fix logfilter event matching with no topic
• 35d5311 go mod tidy
• b3725d9 iadd MultiCallToFilter transform and all that jazz
• a0e3e44 opiniated failure if merge-threshold-block-age is way too low streamingfast/node-manager#23
• 443639f remove firehose flag to gen irr.index, bump libs
• 9465b6b simplify logic in log_address_index

v0.9.0

Changelog

• 930ded3 Add TestBasicLogFilter_GetIndexProvider()
• cde9fb5 Add TestLogAddressIndexProvider_LoadRange()
• 51e6c2a Add TestLogAddressIndexProvider_Matches()
• da5c7c2 Add TestLogAddressIndexProvider_NextMatching()
• aae09d7 Add TestLogAddressIndexProvider_WithinRange()
• 428d1bc Add another case for TestLogAddressIndexProvider_NextMatching()
• cb30473 Add blockNum as log field in findIndexContaining()
• 538a4b5 Add docstring to test fixture
• 3baa7a4 Add guards for GetIndexProvider instantiation
• c4d7b34 Add missing case for TestLogAddressIndexProvider_NextMatching()
• e784dd1 Add more details to docstrings
• b139474 Add stubs for LogAddressIndexProvider
• 5b8c23a Add support for LogAddressIndex (#17)
• bfa3f76 Added .vscode to ignore list
• 4f6db8d Added CI config, added release script and config, updated version printing
• 5f2901e Added Receipt in TrxTrace when creating a new transaction trace
• f285ef5 Added missing Burn balance change reason
• 4efe69e Adjusted instructions
• 4065b9b Adjusted now removed -rpc flags
• a2d34c6 Bump snapshotter
• 6b42571 Bumpd node-manager to latest version
• 042e1c9 Bumped bstream & firehose to latest version
• f02ef9f Bumped bstream + firehose
• fe4994c Bumped dstore to latest version
• 321ad4b Bumped firehose
• f092458 Bumped firehose
• dd5113a Bumped firehose & bstream
• 05e7e7c Bumped firehose to latest version
• a4c342d Bumped node-manager and improved logging when running init command
• d298b78 Bumping to a more recent snapshotter
• 3338f5e Change method signatures for Matches() and NextMatching()
• 10caef1 Cherry-pick stepd/wip
• 7cb85bb Event log transform
• 70fc4aa First pass - add a LogAddressIndexProvider
• e28c17a Fix BasicLogFilter and added testing
• 3acfe9b Fix FindIndexContaining_LoadIndex()
• 8fef352 Fix blockstream flag
• 794e2df Fixed --deep-mind flags that are now prefixed with --firehose-deep-mind
• d11686b Fixed pbcodec.Log#BlockIndex that is wrong when fetched from transactionTraces.calls
• 1bc6aff Fixed generation of proto files
• 0496a3d Fixed go.sum
• 32f88d9 Fixed pb/dfuse not move correctly to pb/sf
• dbb5101 Fixed release script when doing a real non dry run
• 6205bad Fixed some small layout issue in motd
• 9ed057a Flesh-out LogAddressIndexProvider interface methods
• fb13f31 Implement GetIndexProvider for Transform / BasicLogFilter
• 2efada2 Implement GetIndexProvider for Transform / BasicLogFilter (#21)
• 53a4ea6 Increase indexOpsTimeout to 15 seconds
• 9873294 Initial commit
• 67bf8df Marshal / Unmarshal a LogAddressIndex from dstore
• e487c01 Marshal index file and write-out to dstore
• bb7e801 Merge branch 'develop' into stepd/irreversibility_idx
• b2b0215 Merge branch 'stepdrename' into develop
• 6f0aaf1 Merge pull request #14 from streamingfast/stepd/irreversibility_idx
• acd0e9b Merge pull request #19 from streamingfast/froch/wip
• 6bc5ef8 Moved check merged blocks logic to a shared library sf-tools
• 3aeb546 Moved versionning stuff into cli package
• 799676d Re-added Matches and Call Matches implementation
• 1a90c1b Re-generated codec protobuf
• 520bedb Reduce FindIndexContaining_LoadIndex() to a single test case
• ced9693 Remove NewLogAddressSignatureIndex()
• ac671a3 Remove deprecated comments relative to IndexProvider interface
• c1d7b1c Remove vestigial len() check in test fixture
• 433d127 Renamed deth to codec
• 269b537 Renamed scripts header
• f4c71f9 Replace dfuse-io/rpc by streamingfast/rpc
• 375b7e7 Replacement are now grouped to read better
• e08381a Rework LogAddressIndex
• 8f98820 Separate logAddressIndex and logAddressIndexer
• 859aaff Set require.False(t, done) on NextMatching result
• 1d6a9ad Try something else to remove snapshotter dependency error
• 4acfb1a Turn snapshot without compatible eth support into a debug log
• feab03b Updated logging to latest version
• cbaec85 Updated merger to latest
• 1702100 Updated sf-tools for better hole catching in summary printing
• 2963bbe add 'tools generate-irreversible-index' and parametric bundle-sizes to firehose
• 3aa7bf4 add NewLogADdressIndex
• b8a7b14 add draft implementation of log_address_index
• 3320d36 add logADdressIndex matchingBlocks impl
• d366261 add log_address_index provider from transform
• b3b5b55 add status to trxTrace in decoding proto
• 1cfc5cd add transform/indexes from proto defs
• 7874ab8 adjust merger params bump bstream, relayer, nodemanager
• 84e8452 bump bstream and firehose to feature/better_clone_handling
• b353a6f bump bstream for firehose irrIndex fixes
• 1b60819 bump bstream for firehose typo
• e830545 bump bstream for irreversible index creation on first streamable block
• ea92e86 bump bstream for some cleanups
• dca87fe bump bstream to feature/better_clone_handling
• 101fc38 bump bstream to feature/better_clone_handling
• d827714 bump bstream to feature/better_clone_handling
• 6b3e98c bump bstream to feature/better_clone_handling
• 3ddd9a8 bump bstream to feature/better_clone_handling
• c832acc bump bstream to feature/better_clone_handling
• 9455ccf bump bstream to feature/better_clone_handling
• 6f4aa03 bump bstream to feature/better_clone_handling
• 541cbf9 bump bstream to fix firehose CURSOR issues (cursor without lib)
• 0f57a42 bump bstream to fix last (incorrectly fixed) typo
• 64d1b02 bump bstream to fix new memoize/clone, add ATM cache
• 80da1bd bump bstream to fix/index-perf, adjust logaddr_idx
• 4b6f143 bump bstream to fix/index-perf, adjust logaddr_idx (#22)
• d8da678 bump bstream to hotfix atm filesource
• e570734 bump bstream to merged version of PR
• 9d1877d bump bstream, fix atm panic
• cb0ec8b bump bstream, fix some startblocks etc.
• ac0e774 bump bstream, no ProtoGenesisBlk, fixed cursors
• ee03862 bump bstream,firehose, add irreversible_blocks_index partial impl.
• df9c41f bump bstream/firehose after merge
• d4e2b1e bump bstream/firehose irreversibility_idx
• 14d6660 bump firehose and bstream to latest develop
• fb3c9c6 bump firehose for transforms/preproc fix
• 26c45b0 bump firehose, bstream
• 100a2a4 bump merger
• 598b552 bump merger and bstream
• 65e0b32 bump merger fix toOneBlockFile error handling
• c66cbf7 bump merger merged file upload fix
• bb6784b bump merger to drift metrics
• 53df86d bump merger to fix bootstrap
• 22f9145 bump merger with better log
• 1fce895 bump merger with bootstrap from merged files
• e3f2480 bump merger with fixed bootstrap
• dde40e9 bump merger with new bootstrap from lib
• 848d762 bump pbgo and bstream, proto rename from dfuse to sf
• 53b3c64 continue prototype log address index
• 299415e fix WithinRange inefficiencies
• d95a4eb fix log_filter to use Receipt and skip failed logs
• 64b38bc implement LogAddressIndexProvider.findIndexContaining()
• 8f8e350 implement LogAddressIndexProvider.loadIndex()
• 9c1bea3 log in log_address_indexer tool
• 81bdc9a logAddrIdxProvider fix New() assumptions
• d166d6d modify log_address_indexer to work with ethblock not bstreamblock
• 5f4c1dc rebase on develop with transforms
• 775af98 remove admin,debug rpc on default geth flags, nousb and lachesis references
• cbfb9ee remove unsued merger flag 'merger-max-fixable-fork'
• a5c33a2 rename proto index LogAddressMethod to LogAddressSignature
• 90e77d0 wip, debugging test cases with eth blocks
• 8095aac work in progress for filter fix, does not compile