Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add EVM Gateway node ops section and main page #1035

Merged
merged 33 commits into from
Dec 17, 2024
Merged
Show file tree
Hide file tree
Changes from 25 commits
Commits
Show all changes
33 commits
Select commit Hold shift + click to select a range
1c2c855
Add EVm Gateway node ops section and mainpage
franklywatson Dec 10, 2024
05015b9
Add optional docker run args
franklywatson Dec 10, 2024
b4860f1
Typo
franklywatson Dec 10, 2024
5a09a55
Added missing guidance on how to get private key for COA_KEY
franklywatson Dec 10, 2024
24ef40f
Move binary to executable location for systemctl config
franklywatson Dec 10, 2024
c6704ba
Drop duplicate mention of private key extraction
franklywatson Dec 10, 2024
8b08494
Fix title size
franklywatson Dec 10, 2024
9d9f5cb
Tweak navbar position and title
franklywatson Dec 10, 2024
29142a1
Fix naming of the wrong network
franklywatson Dec 10, 2024
b1c61d0
Add troubleshooting items
franklywatson Dec 10, 2024
8008785
Add missing bare metal startup guidance
franklywatson Dec 10, 2024
dd26cd7
Fix title case
franklywatson Dec 10, 2024
7e79976
Fix broken tab
franklywatson Dec 10, 2024
3dce83d
Made comment consistent
franklywatson Dec 10, 2024
be2eed2
Minor tidyup
franklywatson Dec 10, 2024
872d9db
Tidy up troublshooting
franklywatson Dec 10, 2024
f4b941f
Minor fixes
franklywatson Dec 10, 2024
805ac5f
remove redundant line
franklywatson Dec 10, 2024
6b1c620
Make tab titles less confusing
franklywatson Dec 10, 2024
3648e40
Add tab to use Docker image from container registry
franklywatson Dec 10, 2024
78ad0c9
Add comment to run docker images that was pulled for registry
franklywatson Dec 10, 2024
de64fb0
Link to container registry
franklywatson Dec 10, 2024
4733bed
Update docs/networks/node-ops/access-onchain-data/evm-gateway/evm-gat…
franklywatson Dec 11, 2024
02e435e
Update wording in opening description
franklywatson Dec 11, 2024
1b9ddc5
Add guidance to obtain flow cli
franklywatson Dec 11, 2024
afedece
Restore missing shell clause closure
franklywatson Dec 11, 2024
7ff7243
Minor nits
franklywatson Dec 11, 2024
83b8217
Add notice about using old versions of data dir
franklywatson Dec 12, 2024
7bcfae2
Fix mainnet height
franklywatson Dec 13, 2024
0e4952a
Add rate limits
franklywatson Dec 13, 2024
717eb72
Merge remote-tracking branch 'origin/jp/evm-node-ops-improvements' in…
franklywatson Dec 13, 2024
ec3e521
Add new config param
franklywatson Dec 17, 2024
abc1c5a
Fix naming of config for KMS key
franklywatson Dec 17, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 6 additions & 6 deletions docs/evm/using.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -96,7 +96,7 @@ To use the Flow Wallet Chrome extension:
| [eth_getBlockByNumber] | ✅ |
| [eth_getBlockTransactionCountByHash] | ✅ |
| [eth_getBlockTransactionCountByNumber] | ✅ |
| eth_getBlockReceipts | ✅ |
| [eth_getBlockReceipts] | ✅ |
| [eth_getCode] | ✅ |
| [eth_getFilterChanges] | ✅ |
| [eth_getFilterLogs] | ✅ |
Expand All @@ -121,11 +121,11 @@ To use the Flow Wallet Chrome extension:
| [eth_signTransaction] | 🚧 | Unsupported |
| [eth_syncing] | ✅ |
| [eth_uninstallFilter] | ✅ |
| eth_maxPriorityFeePerGas | ✅ |
| eth_feeHistory | ✅ |
| debug_traceTransaction | ✅ |
| debug_traceBlockByNumber | ✅ |
| debug_traceBlockByHash | ✅ |
| [eth_maxPriorityFeePerGas] | ✅ |
| [eth_feeHistory] | ✅ |
| [debug_traceTransaction] | ✅ |
| [debug_traceBlockByNumber] | ✅ |
| [debug_traceBlockByHash] | ✅ |

**Legend**: ❌ = not supported. 🚧 = work in progress. ✅ = supported.

Expand Down
6 changes: 6 additions & 0 deletions docs/networks/flow-networks/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,12 @@ sidebar_position: 1

## About Flow Networks

:::note

This page provides information on Flow network RPCs. EVM on Flow network RPCs can be found [here](../evm/networks)

:::

In addition to Mainnet, developers have access to the Testnet environment, which serves as an essential testing ground for applications and smart contracts prior to their deployment on Mainnet. This ensures that any potential issues can be identified and resolved in a controlled setting, mitigating risks associated with live deployment.

Furthermore, during network upgrades, Testnet receives updates ahead of Mainnet. This preemptive update process allows developers to comprehensively test their apps against the latest versions of the nodes, enhancements to the Cadence programming language, and core contract upgrades. This strategy guarantees that when these updates are eventually applied to Mainnet, applications and smart contracts will operate seamlessly, enhancing overall network stability and user experience.
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
label: EVM Gateway
Original file line number Diff line number Diff line change
@@ -0,0 +1,323 @@
---
title: Setting up an EVM Gateway node
sidebar_label: EVM Gateway Setup
sidebar_position: 3
---

import Tabs from '@theme/Tabs';
import TabItem from '@theme/TabItem';

This guide is for running the [EVM Gateway](https://github.com/onflow/flow-evm-gateway) node on Flow. The EVM Gateway implements the
[Ethereum JSON-RPC specification](https://ethereum.org/en/developers/docs/apis/json-rpc/) and is the only node type which accepts EVM
client connections.

The EVM Gateway consumes Flow protocol state from the configured Flow Access Node and persists the indexed EVM state locally to service EVM client requests. It submits EVM transactions it receives into the Flow network, wrapped in a Cadence transaction, and mutating EVM state when executed. Non-mutating RPC methods only query the local state index of the gateway and are never forwarded to Access Nodes. It does not participate in the block production process and requires no stake.

## Who Should Run an EVM Gateway

The EVM Gateway can serve as both a dedicated private RPC endpoint and a performance scaling solution, offering similar capabilities
to centralized middleware providers like Infura, Alchemy, etc at a fraction of the cost. This is because EVM Gateway nodes connect
directly to the Flow network with no middle layer in between.

Applications which generate high call volumes to the JSON-RPC and which may have hit rate limits can benefit from running their
own gateway. Self-hosted gateways are dedicated to the operator, enabling the removal of rate limits. Self-hosted gateways
connect directly to your chosen Access Node, which you can also run if desired.

## Hardware specifications

TBD

# How To Run EVM Gateway

## Step 1 - Account Creation

The EVM Gateway's role in mediating EVM transactions over to Cadence is how it accrues fees from handling client transactions. Since
the gateway submits Cadence transactions wrapping EVM transaction payloads to the Flow Access Node the transaction fee for that must
be paid by the EVM Gateway.

The account used for funding gateway Cadence transactions must be a COA, not an EOA. `--coa-address` is configured with the Cadence address
of the COA account and the `--coa-key` must belong to the same account. The `--coinbase` account accrues EVM Gateway fees from EVM client
transactions and can be either an EVM EOA or COA address.

It is acceptable to create a single Cadence account for the COA and use the EVM address associated with that for the COINBASE address.

### Create Flow account to use for COA

If you don't already have a Flow account you will need to create one.

<Tabs>
<TabItem value="mainnet" label="Mainnet">

1. Install [Flow Wallet](https://wallet.flow.com/)
2. Once installed you will be able to copy the wallet address, similar to _0x1844efeb3fef2242_
3. Obtain account private key from
<pre>Settings -> Account List -> Choose Main account -> Private Key -> [Password prompt]</pre>
4. Ensure the wallet is funded from a CEX or other wallet

</TabItem>
<TabItem value="testnet" label="Testnet">
Install [Flow CLI](https://developers.flow.com/tools/flow-cli/install) if not already installed.

```bash
flow keys generate
This will output something similar to:

franklywatson marked this conversation as resolved.
Show resolved Hide resolved
```bash
🔴️ Store private key safely and don't share with anyone!
Private Key 3cf8334d.....95c3c54a28e4ad1
Public Key 33a13ade6....85f1b49a197747
Mnemonic often scare peanut ... boil corn change
Derivation Path m/44'/539'/0'/0/0
Signature Algorithm ECDSA_P256
```

Visit https://faucet.flow.com/, and use the generated `Public Key`, to create and fund your Flow testnet account.

</TabItem>
</Tabs>

## Step 2 - Build the gateway

To run EVM Gateway on bare metal or in a VM without the use of docker, select the '_Build from source_' tab otherwise refer to the
'_Build using Docker_' tab.

<Tabs>
<TabItem value="source-build" label="Build from source">

This will build the EVM gateway binary from source.

```bash
git clone https://github.com/onflow/flow-evm-gateway.git

cd flow-evm-gateway
git checkout $(curl -s https://api.github.com/repos/onflow/flow-evm-gateway/releases/latest | jq -r .tag_name)
CGO_ENABLED=1 go build -o evm-gateway cmd/main/main.go
chmod a+x evm-gateway
mv evm-gateway /usr/bin
```

</TabItem>
<TabItem value="docker-build" label="Build using Docker">

```bash
git clone https://github.com/onflow/flow-evm-gateway.git

cd flow-evm-gateway
git checkout $(curl -s https://api.github.com/repos/onflow/flow-evm-gateway/releases/latest | jq -r .tag_name)
make docker-build
```

</TabItem>
<TabItem value="docker-install" label="Use Docker registry image">

Registry versions available for download can be found [here](https://console.cloud.google.com/artifacts/docker/dl-flow-devex-production/us-west1/development/flow-evm-gateway).
```bash
docker pull us-west1-docker.pkg.dev/dl-flow-devex-production/development/flow-evm-gateway:${VERSION}
```

</TabItem>
</Tabs>

## Step 3 - Start Your Node

Operators will need to refer to the gateway [configuration flags](https://github.com/onflow/flow-evm-gateway?tab=readme-ov-file#configuration-flags) and make adjustments that align with the desired
deployment topology.

### EVM Coinbase address

If this is your first time setting up the gateway we need to ensure that an EVM COA or EOA address is available to configure the `COINBASE`. This account
can be an account created using Metamask or other web3.js wallet, or otherwise can be the EVM address corresponding to the Flow Wallet COA account created above.

If you haven't already got an EVM address and you have the COA account created by Flow Wallet above then follow the steps below:

* Click top left burger icon to show current profile
* Click 'Enable the path to EVM on Flow' button
* Your EVM account will now be available to use in the left nav account view
* When you switch to that account you can obtain its EVM address

### COA Address and Key

COA address and private key is configured for `--coa-address` & `--coa-key` configuration flags. If running multiple EVM Gateway hosts it is standard to
share the same COA address and key across _n_ hosts.

### Run the gateway

Ensure that the following ENV variables have been set. Add/update as required if your configuration differs from those listed.

```bash
# Set required environment variables
export ACCESS_NODE_GRPC_HOST="access.mainnet.nodes.onflow.org:9000" # or access.devnet.nodes.onflow.org:9000 for testnet
export FLOW_NETWORK_ID="flow-mainnet" # or flow-testnet
export INIT_CADENCE_HEIGHT="88226267" # 211176670 for testnet
export COINBASE="${EVM_ADDRESS_WITHOUT_0x}"
export COA_ADDRESS="${CADENCE_ACCOUNT_ADDRESS_WITHOUT_0x}"
export COA_KEY="${CADENCE_ACCOUNT_PRIVATE_KEY_WITHOUT_0x}"
export GAS_PRICE="100"
```

<Tabs>
<TabItem value="source-build" label="Run from binary">

**Create EVM Gateway service**

```bash
sudo tee <<EOF >/dev/null /etc/systemd/system/gateway.service
[Unit]
Description=Gateway daemon
After=network-online.target

[Service]
User=$USER
ExecStart=/usr/bin/evm-gateway \
--access-node-grpc-host=$ACCESS_NODE_GRPC_HOST \
--flow-network-id=$FLOW_NETWORK_ID \
--init-cadence-height=$INIT_CADENCE_HEIGHT \
--ws-enabled=true \
--coinbase=$COINBASE \
--coa-address=$COA_ADDRESS \
--coa-key=$COA_KEY \
--rate-limit=9999999 \
--rpc-host=0.0.0.0 \
--gas-price=$GAS_PRICE
franklywatson marked this conversation as resolved.
Show resolved Hide resolved
Restart=always
RestartSec=3
LimitNOFILE=4096

[Install]
WantedBy=multi-user.target
EOF

cat /etc/systemd/system/gateway.service
sudo systemctl enable gateway
```

**Start all services**

```bash
sudo systemctl daemon-reload
sudo systemctl restart access-node
sudo systemctl restart gateway
```

**Check logs**

```bash
# change log settings to persistent if not already
sed -i 's/#Storage=auto/Storage=persistent/g' /etc/systemd/journald.conf
sudo systemctl restart systemd-journald

journalctl -u gateway.service -f -n 100
```

</TabItem>
<TabItem value="docker-build" label="Run with Docker">

It may be necessary to make local changes to the `docker-run` target to add params which are needed for your requirements. If you pulled a
specific image from the gateway container registry ensure that the `$VERSION` environment variable is set to the same as the image version
you pulled.

```bash
cd flow-evm-gateway
make docker-run
```
Additional options are available as follows

```bash
DOCKER_RUN_DETACHED=true
DOCKER_HOST_MOUNT=[host mount directory]
DOCKER_HOST_PORT=[desired port to expose on host]

# Example usage

make DOCKER_RUN_DETACHED=true DOCKER_HOST_PORT=1234 DOCKER_HOST_MOUNT=/my/host/dir docker-run
```

</TabItem>
</Tabs>

### Startup bootstrap indexing

Once your EVM Gateway is up and running you will see it indexing the Flow network which was configured. At the present time this
is a lengthy process (possibly 1-3 days, depending on CPU core count) during which time the gateway will not respond to queries.
Once the data is fully indexed the gateway can serve requests to clients.

To speed up gateway setup we recommend backing up the `/${GATEWAY_HOME_DIR}/data` directory to use when creating additional nodes
using the same release version. We are currently working on an export/import feature that will enable gateway operators to
store state snapshots to bootstrap newly created nodes without the delay.

### Account and Key Management

EVM Gateway allows for Google and AWS Key Management Service (KMS) setup, which is the recommended way of setting up the gateway
for live networks. We recommend creating multiple KMS keys for the same Flow account (ideally 10 or more), how many depends on
the desired transaction throughput, since the keys are used in rotation when submitting the transactions. If too few keys
are configured it may result in sequence number collisions if the same key is used concurrently by multiple EVM client requests.

### KMS Configuration
```
--coa-cloud-kms-project-id=your-project-kms-id \
--coa-cloud-kms-location-id=global \
--coa-cloud-kms-key-ring-id=your-project-kms-key-ring-id \
--coa-cloud-kms-keys=example-gcp-kms1@1,example-gcp-kms2@1 \
franklywatson marked this conversation as resolved.
Show resolved Hide resolved
```

### Monitoring and Metrics

The EVM Gateway reports Prometheus metrics which are a way to monitor the gateway's availability and progress. The database folder
size may also need to be monitored to prevent disk full issues.

**Metric labels**
```bash
evm_gateway_api_errors_total # Total count of API errors for period
evm_gateway_api_request_duration_seconds_bucket # Histogram metric buckets for API request durations
evm_gateway_api_request_duration_seconds_count # Histogram metric API request count for period
evm_gateway_api_request_duration_seconds_sum # Histogram metric API request sum of values for period
evm_gateway_api_server_panics_total # Total count of server panics for period
evm_gateway_blocks_indexed_total # Total count of EVM blocks indexed
evm_gateway_cadence_block_height # Cadence block height
evm_gateway_evm_account_interactions_total # Count of unique accounts observed for period
evm_gateway_evm_block_height # EVM block height
evm_gateway_operator_balance # Gateway node COA operator account balance
evm_gateway_trace_download_errors_total # Total count of trace download errors
evm_gateway_txs_indexed_total # Total count of indexed transactions
```

Alerts are recommended to be configured on server panics, low operator balance, and disk usage metrics.

### Metrics port
```
--metrics-port 8080 \
```
### Node Status

For basic node status or keepalive monitoring we recommend automated checks on the following monotonically increasing counter:
```
curl -s -XPOST 'your-evm-gw-host:8545' --header 'Content-Type: application/json' --data-raw '{"jsonrpc":"2.0","method":"eth_blockNumber","params":[],"id":1}' | jq -r '.result' | xargs printf "%d\n"
10020239
```

## Troubleshooting

### State stream configuration

If you are running an Access Node on the same logical host as the EVM Gateway you may see ehe following log entries.

```bash
failure in event subscription at height ${INIT-CADENCE-HEIGHT}, with: recoverable: disconnected: error receiving event: rpc error: code = Unimplemented desc = unknown service flow.executiondata.ExecutionDataAPI”
```
```bash
component execution data indexer initialization failed: could not verify checkpoint file: could not find expected root hash e6d4f4c755666c21d7456441b4d33d3521e5e030b3eae391295577e9130fd715 in checkpoint file which contains: [e10d3c53608a1f195b7969fbc06763285281f64595be491630a1e1bdfbe69161]
```

To resolve this configure `--state-stream-addr` to use the same address/port combination which is set for Access Node `--rpc-addr`.
This is required by the gateway to allow both the streaming and non-streaming APIs to query using the same connection.

### Access Node not fully synced

The following log entry will occur when the EVM Gateway attempts to sync with the Access Node but it has not yet synced up to latest block
```bash
failure in event subscription at height ${INIT-CADENCE-HEIGHT}, with: recoverable: disconnected: error receiving event: rpc error: code = FailedPrecondition desc = could not get start height: failed to get lowest indexed height: index not initialized
```

## FAQs

TBD
Loading