Skip to content

Releases: streamingfast/firehose-ethereum

v2.0.1

05 Dec 18:36
@sduchesneau sduchesneau
v2.0.1
8bf0668
Compare
Choose a tag to compare
v2.0.1
  • Fixed SF_TRACING feature (regression broke the ability to specify a tracing endpoint)
  • Fixed substreams GRPC/Connect error codes not propagating correctly
  • Firehose connections rate-limiting will now force an (increased) delay of between 1 and 4 seconds (random value) before refusing a connection when under heavy load
Assets 7
Loading

v2.0.0

16 Nov 16:40
@sduchesneau sduchesneau
v2.0.0
b067b0a
Compare
Choose a tag to compare
v2.0.0

Major release v2 Highlights

This releases refactor firehose-ethereum repository to use the common shared Firehose Core library (https://github.com/streamingfast/firehose-core) that every single Firehose supported chain should use and follow.

Both at the data level and gRPC level, there is no changes in behavior to all core components which are reader-node, merger, relayer, firehose, substreams-tier1 and substreams-tier2.

A lot of changes happened at the operators level however and some superflous mode have been removed, especially around the reader-node application. The full changes is listed below, operators should review thoroughly the changelog.

Important

It's important to emphasis that at the data level, nothing changed, so reverting to 1.4.22 in case of a problem is quite easy and no special data migration is required outside of changing back to the old set of flags that was used before.

Operators

You will find below the detailed upgrade procedure for the configuration file operators usually use. If you are using the flags based approach, simply update the corresponding flags.

Important

We have had reports of older versions of this software creating corrupted merged-blocks-files (with duplicate or out-of-bound blocks). This release adds additional validation of merged-blocks to prevent serving duplicate blocks from the firehose or substreams service. This may cause service outage if you have produced those blocks or downloaded them from another party who was affected by this bug.
See the Finding and fixing corrupted merged-blocks-files to see how you can prevent service outage.

Quick Upgrade

Here a bullet list for upgrading your instance, we still recommend to fully read each section below, the list here can serve as a check list. The list below is done in such way that you get back the same "instance" as before. The listening addresses changes can be omitted as long as you update other tools to account for the port changes list your load balancer.

  • Add config config-file: ./sf.yaml if not present already

  • Add config data-dir: ./sf-data if not present already

  • Rename config verbose to log-verbosity if present

  • Add config common-blocks-cache-dir: ./sf-data/blocks-cache if not present already

  • Remove config common-chain-id if present

  • Remove config common-deployment-id if present

  • Remove config common-network-id if present

  • Add config common-live-blocks-addr: :13011 if not present already

  • Add config relayer-grpc-listen-addr: :13011 if common-live-blocks-addr has been added in previous step

  • Add config reader-node-grpc-listen-addr: :13010 if not present already

  • Add config relayer-source: :13010 if reader-node-grpc-listen-addr has been added in previous step

  • Remove config reader-node-enforce-peers if present

  • Remove config reader-node-log-to-zap if present

  • Remove config reader-node-ipc-path if present

  • Remove config reader-node-type if present

  • Replace config reader-node-arguments: +--<flag1> --<flag2> ... by reader-node-arguments: --networkid=<network-id> --datadir={node-data-dir} --port=30305 --http --http.api=eth,net,web3 --http.port=8547 --http.addr=0.0.0.0 --http.vhosts=* --firehose-enabled --<flag1> --<flag2> ...

    [!NOTE]
    The <network-id> is dynamic and should be replace with a literal value like 1 for Ethereum Mainnet. The {node-data-dir} value is actually a templating value that is going o be resolved for you (resolves to value of config reader-node-data-dir).!

    [!IMPORTANT]
    Ensure that --firehose-enabled is part of the flag! Moreover, tweak flags to avoid repetitions if your were overriding some of them.

  • Remove node under start: args: list

  • Add config merger-grpc-listen-addr: :13012 if not present already

  • Add config firehose-grpc-listen-addr: :13042 if not present already

  • Add config substreams-tier1-grpc-listen-addr: :13044 if not present already

  • Add config substreams-tier1-grpc-listen-addr: :13044 if not present already

  • Add config substreams-tier2-grpc-listen-addr: :13045 if not present already

  • Add config substreams-tier1-subrequests-endpoint: :13045 if substreams-tier1-grpc-listen-addr has been added in previous step

  • Replace config combined-index-builder to index-builder under start: args: list

  • Rename config common-block-index-sizes to common-index-block-sizes if present

  • Rename config combined-index-builder-grpc-listen-addr to index-builder-grpc-listen-addr if present

  • Add config index-builder-grpc-listen-addr: :13043 if you didn't have combined-index-builder-grpc-listen-addr previously

  • Rename config combined-index-builder-index-size to index-builder-index-size if present

  • Rename config combined-index-builder-start-block to index-builder-start-block if present

  • Rename config combined-index-builder-stop-block to index-builder-stop-block if present

  • Replace any occurrences of {sf-data-dir} to {data-dir} in any of your configuration values if present

Common Changes

  • The default value for config-file changed from sf.yaml to firehose.yaml. If you didn't had this flag defined and wish to keep the old default, define config-file: sf.yaml.

  • The default value for data-dir changed from sf-data to firehose-data. If you didn't had this flag defined before, you should either move sf-data to firehose-data or define data-dir: sf-data.

    [!NOTE]
    This is an important change, forgetting to change it will change expected locations of data leading to errors or wrong data.

  • Deprecated The {sf-data-dir} templating argument used in various flags to resolve to the --data-dir=<location> value has been deprecated and should now be simply {data-dir}. The older replacement is still going to work but you should replace any occurrences of {sf-data-dir} in your flag definition by {data-dir}.

  • The default value for common-blocks-cache-dir changed from {sf-data-dir}/blocks-cache to file://{data-dir}/storage/blocks-cache. If you didn't had this flag defined and you had common-blocks-cache-enabled: true, you should define common-blocks-cache-dir: file://{data-dir}/blocks-cache.

  • The default value for common-live-blocks-addr changed from :13011 to :10014. If you didn't had this flag defined and wish to keep the old default, define common-live-blocks-addr: 13011 and ensure you also modify relayer-grpc-listen-addr: :13011 (see next entry for details).

  • The Go module github.com/streamingfast/firehose-ethereum/types has been removed, if you were depending on github.com/streamingfast/firehose-ethereum/types in your project before, depend directly on github.com/streamingfast/firehose-ethereum instead.

    [!NOTE]
    This will pull much more dependencies then before, if you're reluctant of such additions, talk to us on Discord and we can offer alternatives depending on what you were using.

  • The config value verbose has been renamed to log-verbosity keeping the same semantic and default value as before

    [!NOTE]
    The short flag version is still -v and can still be provided multiple times like -vvvv.

App reader-node changes

This change will impact all operators currently running Firehose on Ethereum so it's important to pay attention to the upgrade procedure below, if you are unsure of something, reach to us on Discord.

Before this release, the reader-node app was managing for you a portion of the reader-node-arguments configuration value, prepending some arguments that would be passed to geth when invoking it, the list of arguments that were automatically provided before:

  • --networkid=<value of config value 'common-network-id'>
  • --datadir=<value of config value 'reader-node-data-dir'>
  • --ipcpath=<value of config value 'reader-node-ipc-path'>
  • --port=30305
  • --http
  • --http.api=eth,net,web3
  • --http.port=8547
  • --http.addr=0.0.0.0
  • --http.vhosts=*
  • --firehose-enabled

We have now removed those magical additions and operators are now responsible of providing the flags they required to properly run a Firehose-enabled native geth node. The + sign that was used to append/override the flags has been removed also since no default additions is performed, the + was now useless. To make some flag easier to define and avoid repetition, a few templating variable can be used within the reader-node-arguments value:

  • {data-dir} The current data-dir path defined by the config value data-dir
  • {node-data-dir} The node data dir path defined by the flag reader-node-data-dir
  • {hostname} The machine's hostname
  • {start-block-num} The resolved start block number defined by the flag reader-node-start-block-num (can be overwritten)
  • {stop-block-num} The stop block number defined by the flag reader-node-stop-block-num

As an example, if you provide the config value reader-node-data-dir=/var/geth for example, then you could use reader-node-arguments: --datadir={node-data-dir} and that would resolve to reader-node-arguments: --datadir=/var/geth for you.

Note

The reader-node-arguments is a string that is parsed using Shell word splitting rules which means for example that double quotes are supported like --datadir="/var/with space/path" and the argument will be correctly accepted. We use https://github.com/kballard/go-shellquote as your parsing library.

We also removed the following reader-node configuration value:

  • reader-node-type (No replacement needed, just remove it)
  • reader-node-ipc-path (If you were using that, define it manually using geth flag --ipcpath=...)
  • reader-node-enforce-peers (If you were using that, use a geth config file to add static peers to your...
Read more
Assets 8
Loading

v2.0.0-rc.1

10 Nov 16:23
@sduchesneau sduchesneau
v2.0.0-rc.1
ff0760b
Compare
Choose a tag to compare
v2.0.0-rc.1

Highlights

This releases refactor firehose-ethereum repository to use the common shared Firehose Core library (https://github.com/streamingfast/firehose-core) that every single Firehose supported chain should use and follow.

Both at the data level and gRPC level, there is no changes in behavior to all core components which are reader-node, merger, relayer, firehose, substreams-tier1 and substreams-tier2.

A lot of changes happened at the operators level however and some superflous mode have been removed, especially around the reader-node application. The full changes is listed below, operators should review thoroughly the changelog.

Important

It's important to emphasis that at the data level, nothing changed, so reverting to 1.4.22 in case of a problem is quite easy and no special data migration is required outside of changing back to the old set of flags that was used before.

Operators

You will find below the detailed upgrade procedure for the configuration file operators usually use. If you are using the flags based approach, simply update the corresponding flags.

Important

We have had reports of older versions of this software creating corrupted merged-blocks-files (with duplicate or out-of-bound blocks). This release adds additional validation of merged-blocks to prevent serving duplicate blocks from the firehose or substreams service. This may cause service outage if you have produced those blocks or downloaded them from another party who was affected by this bug.
See the Finding and fixing corrupted merged-blocks-files to see how you can prevent service outage.

Quick Upgrade

Here a bullet list for upgrading your instance, we still recommend to fully read each section below, the list here can serve as a check list. The list below is done in such way that you get back the same "instance" as before. The listening addresses changes can be omitted as long as you update other tools to account for the port changes list your load balancer.

  • Add config config-file: ./sf.yaml if not present already

  • Add config data-dir: ./sf-data if not present already

  • Rename config verbose to log-verbosity if present

  • Add config common-blocks-cache-dir: ./sf-data/blocks-cache if not present already

  • Remove config common-chain-id if present

  • Remove config common-deployment-id if present

  • Remove config common-network-id if present

  • Add config common-live-blocks-addr: :13011 if not present already

  • Add config relayer-grpc-listen-addr: :13011 if common-live-blocks-addr has been added in previous step

  • Add config reader-node-grpc-listen-addr: :13010 if not present already

  • Add config relayer-source: :13010 if reader-node-grpc-listen-addr has been added in previous step

  • Remove config reader-node-enforce-peers if present

  • Remove config reader-node-log-to-zap if present

  • Remove config reader-node-ipc-path if present

  • Remove config reader-node-type if present

  • Replace config reader-node-arguments: +--<flag1> --<flag2> ... by reader-node-arguments: --networkid=<network-id> --datadir={node-data-dir} --port=30305 --http --http.api=eth,net,web3 --http.port=8547 --http.addr=0.0.0.0 --http.vhosts=* --firehose-enabled --<flag1> --<flag2> ...

    [!NOTE]
    The <network-id> is dynamic and should be replace with a literal value like 1 for Ethereum Mainnet. The {node-data-dir} value is actually a templating value that is going o be resolved for you (resolves to value of config reader-node-data-dir).!

    [!IMPORTANT]
    Ensure that --firehose-enabled is part of the flag! Moreover, tweak flags to avoid repetitions if your were overriding some of them.

  • Remove node under start: args: list

  • Add config merger-grpc-listen-addr: :13012 if not present already

  • Add config firehose-grpc-listen-addr: :13042 if not present already

  • Add config substreams-tier1-grpc-listen-addr: :13044 if not present already

  • Add config substreams-tier1-grpc-listen-addr: :13044 if not present already

  • Add config substreams-tier2-grpc-listen-addr: :13045 if not present already

  • Add config substreams-tier1-subrequests-endpoint: :13045 if substreams-tier1-grpc-listen-addr has been added in previous step

  • Replace config combined-index-builder to index-builder under start: args: list

  • Rename config common-block-index-sizes to common-index-block-sizes if present

  • Rename config combined-index-builder-grpc-listen-addr to index-builder-grpc-listen-addr if present

  • Add config index-builder-grpc-listen-addr: :13043 if you didn't have combined-index-builder-grpc-listen-addr previously

  • Rename config combined-index-builder-index-size to index-builder-index-size if present

  • Rename config combined-index-builder-start-block to index-builder-start-block if present

  • Rename config combined-index-builder-stop-block to index-builder-stop-block if present

  • Replace any occurrences of {sf-data-dir} to {data-dir} in any of your configuration values if present

Common Changes

  • The default value for config-file changed from sf.yaml to firehose.yaml. If you didn't had this flag defined and wish to keep the old default, define config-file: sf.yaml.

  • The default value for data-dir changed from sf-data to firehose-data. If you didn't had this flag defined before, you should either move sf-data to firehose-data or define data-dir: sf-data.

    [!NOTE]
    This is an important change, forgetting to change it will change expected locations of data leading to errors or wrong data.

  • Deprecated The {sf-data-dir} templating argument used in various flags to resolve to the --data-dir=<location> value has been deprecated and should now be simply {data-dir}. The older replacement is still going to work but you should replace any occurrences of {sf-data-dir} in your flag definition by {data-dir}.

  • The default value for common-blocks-cache-dir changed from {sf-data-dir}/blocks-cache to file://{data-dir}/storage/blocks-cache. If you didn't had this flag defined and you had common-blocks-cache-enabled: true, you should define common-blocks-cache-dir: file://{data-dir}/blocks-cache.

  • The default value for common-live-blocks-addr changed from :13011 to :10014. If you didn't had this flag defined and wish to keep the old default, define common-live-blocks-addr: 13011 and ensure you also modify relayer-grpc-listen-addr: :13011 (see next entry for details).

  • The Go module github.com/streamingfast/firehose-ethereum/types has been removed, if you were depending on github.com/streamingfast/firehose-ethereum/types in your project before, depend directly on github.com/streamingfast/firehose-ethereum instead.

    [!NOTE]
    This will pull much more dependencies then before, if you're reluctant of such additions, talk to us on Discord and we can offer alternatives depending on what you were using.

  • The config value verbose has been renamed to log-verbosity keeping the same semantic and default value as before

    [!NOTE]
    The short flag version is still -v and can still be provided multiple times like -vvvv.

App reader-node changes

This change will impact all operators currently running Firehose on Ethereum so it's important to pay attention to the upgrade procedure below, if you are unsure of something, reach to us on Discord.

Before this release, the reader-node app was managing for you a portion of the reader-node-arguments configuration value, prepending some arguments that would be passed to geth when invoking it, the list of arguments that were automatically provided before:

  • --networkid=<value of config value 'common-network-id'>
  • --datadir=<value of config value 'reader-node-data-dir'>
  • --ipcpath=<value of config value 'reader-node-ipc-path'>
  • --port=30305
  • --http
  • --http.api=eth,net,web3
  • --http.port=8547
  • --http.addr=0.0.0.0
  • --http.vhosts=*
  • --firehose-enabled

We have now removed those magical additions and operators are now responsible of providing the flags they required to properly run a Firehose-enabled native geth node. The + sign that was used to append/override the flags has been removed also since no default additions is performed, the + was now useless. To make some flag easier to define and avoid repetition, a few templating variable can be used within the reader-node-arguments value:

  • {data-dir} The current data-dir path defined by the config value data-dir
  • {node-data-dir} The node data dir path defined by the flag reader-node-data-dir
  • {hostname} The machine's hostname
  • {start-block-num} The resolved start block number defined by the flag reader-node-start-block-num (can be overwritten)
  • {stop-block-num} The stop block number defined by the flag reader-node-stop-block-num

As an example, if you provide the config value reader-node-data-dir=/var/geth for example, then you could use reader-node-arguments: --datadir={node-data-dir} and that would resolve to reader-node-arguments: --datadir=/var/geth for you.

Note

The reader-node-arguments is a string that is parsed using Shell word splitting rules which means for example that double quotes are supported like --datadir="/var/with space/path" and the argument will be correctly accepted. We use https://github.com/kballard/go-shellquote as your parsing library.

We also removed the following reader-node configuration value:

  • reader-node-type (No replacement needed, just remove it)
  • reader-node-ipc-path (If you were using that, define it manually using geth flag --ipcpath=...)
  • reader-node-enforce-peers (If you were using that, use a geth config file to add static peers to your node, read ...
Read more
Assets 7
Loading

v1.4.22

27 Oct 21:10
@maoueh maoueh
v1.4.22
85cff64
Compare
Choose a tag to compare
v1.4.22

Fixed a regression where reader-node-role default value was changed to dev by mistake, this release fixes by reverting back to the default geth value.

You should skip v1.4.21 and upgrade straight to v1.4.22

Assets 7
Loading

v1.4.21

27 Oct 11:44
@colindickson colindickson
v1.4.21
eba5708
Compare
Choose a tag to compare
v1.4.21

Bump Substreams to v1.1.20 with a fix for some minor bug fixes related to start block processing.

Assets 7
Loading

v1.4.20

25 Oct 19:11
@sduchesneau sduchesneau
v1.4.20
b017b0c
Compare
Choose a tag to compare
v1.4.20

Added

  • Added tools poll-rpc-blocks command to launch an RPC-based poller that acts as a firehose extractor node, printing base64-encoded protobuf blocks to stdout (used by the 'dev' node-type). It creates "light" blocks, without traces and ordinals.
  • Added --dev flag to the start command to simplify running a local firehose+substreams stack from a development node (ex: Hardhat).
    • This flag overrides the --reader-node-path, instead pointing to the fireeth binary itself.
    • This flag overrides the --reader-node-type, setting it to dev instead of geth.
      This node type has the following default reader-node-arguments: tools poll-rpc-blocks http://localhost:8545 0
    • It also removes node from the list of default apps

Fixed

  • Substreams: fixed metrics calculations (per-module processing-time and external calls were wrong)
  • Substreams: fixed immediate EOF when streaming from block 0 to (unbounded) in dev mode
Assets 7
Loading

v1.4.19

16 Oct 14:20
@colindickson colindickson
v1.4.19
c496607
Compare
Choose a tag to compare
v1.4.19
  • Bumped substreams to v1.1.18 with a regression fix for when a substreams has a start block in the reversible segment
Assets 7
Loading

v1.4.18

10 Oct 17:24
@billettc billettc
v1.4.18
31363d2
This commit was created on GitHub.com and signed with GitHub’s verified signature. The key has expired.
GPG key ID: 4AEE18F83AFDEB23
Expired
Verified
Learn about vigilant mode.
Compare
Choose a tag to compare
v1.4.18
  • Bumped substreams to v1.1.17 with fix missing decrement on metrics substreams_active_requests`
Assets 7
Loading

v1.4.17

10 Oct 16:41
@billettc billettc
v1.4.17
76e189b
Compare
Choose a tag to compare
v1.4.17

Added

The --common-auth-plugin got back the ability to use secret://<expected_secret>?[user_id=<user_id>]&[api_key_id=<api_key_id>] in which case request are authenticated based on the Authorization: Bearer <actual_secret> and continue only if <actual_secret> == <expected_secret>.

Changed

  • Bumped substreams to v1.1.16 with support of metrics substreams_active_requests and substreams_counter
Assets 7
Loading

v1.4.16

29 Sep 15:33
@sduchesneau sduchesneau
v1.4.16
d2f8a08
Compare
Choose a tag to compare
v1.4.16

Operators of Polygon/Mumbai chains

  • If you started reprocessing the blockchain blocks using release v1.4.14 or v1.4.15, you will need to run the following command to fix the blocks affected by another bug:
    fireeth tools fix-polygon-index /your/merged/blocks /temporary/destination 0 48200000 (note that you can run multiple instances of this command in parallel to cover the range of blocks from 0 to current HEAD in smaller chunks)

Fixed

  • Fix another data issue found in polygon blocks: blocks that contain a single "system" transaction have "Index=1" for that transaction instead of "Index=0"
Assets 7
Loading