Configuration file (#131)

# Description

Configuration methodology (via CLI options) has been outgrown with the
number of options. This PR removes all chain / network related
configuration and moves to a configuration file.

# Changes

- [x] Move all configuration for chains to a dedicated file
- [x] Remove unused features (replay tx / replay block)
- [x] Eliminates dynamic download of filter policies (now within
configuration file)
- [x] Update `README.md`

## How to test
1. Adjust the `config.json.example` for suitable configuration
2. Run and verify sync process completes warmup

## Related Issues

Fixes #129

---------

Co-authored-by: Leandro <[email protected]>

README.md

@@ -1,21 +1,21 @@
-# Watch-Tower for Composable CoWs 🐮🎶
+# Watch-Tower for Programmatic Orders 🐮🎶
 
 ## Overview
 
-The [`ComposableCoW`](https://github.com/cowprotocol/composable-cow) conditional order framework requires a watch-tower to monitor the blockchain for new orders, and to post them to the [CoW Protocol `OrderBook` API](https://api.cow.fi/docs/#/). The watch-tower is a standalone application that can be run locally as a script for development, or deployed as a docker container to a server, or dappnode.
+The [programmatic order](https://docs.cow.fi/cow-protocol/concepts/order-types/programmatic-orders) framework requires a watch-tower to monitor the blockchain for new orders, and to post them to the [CoW Protocol `OrderBook` API](https://docs.cow.fi/cow-protocol/reference/apis/orderbook). The watch-tower is a standalone application that can be run locally as a script for development, or deployed as a docker container to a server, or dappnode.
 
 ## Deployment
 
 If running your own watch-tower instance, you will need the following:
 
 - An RPC node connected to the Ethereum mainnet, Gnosis Chain, or Goerli.
-- Internet access to the [CoW Protocol `OrderBook` API](https://api.cow.fi/docs/#/).
+- Internet access to the [CoW Protocol `OrderBook` API](https://docs.cow.fi/cow-protocol/reference/apis/orderbook).
 
 **CAUTION**: Conditional order types may consume considerable RPC calls.
 
 **NOTE**: `deployment-block` refers to the block number at which the **`ComposableCoW`** contract was deployed to the respective chain. This is used to optimise the watch-tower by only fetching events from the blockchain after this block number. Refer to [Deployed Contracts](https://github.com/cowprotocol/composable-cow#deployed-contracts) for the respective chains.
 
-**NOTE**: The `--page-size` option is used to specify the number of blocks to fetch from the blockchain when querying historical events (`eth_getLogs`). The default is `5000`, which is the maximum number of blocks that can be fetched in a single request from Infura. If you are running the watch-tower against your own RPC, you may want to set this to `0` to fetch all blocks in one request, as opposed to paging requests.
+**NOTE**: The `pageSize` option is used to specify the number of blocks to fetch from the blockchain when querying historical events (`eth_getLogs`). The default is `5000`, which is the maximum number of blocks that can be fetched in a single request from Infura. If you are running the watch-tower against your own RPC, you may want to set this to `0` to fetch all blocks in one request, as opposed to paging requests.
 
 
 ### Docker
@@ -31,20 +31,13 @@ As an example, to run the latest version of the watch-tower via `docker`:
 
 ```bash
 docker run --rm -it \
+  -v "$(pwd)/config.json.example:/config.json" \
   ghcr.io/cowprotocol/watch-tower:latest \
   run \
-  --chain-config <rpc>,<deployment-block> \
-  --page-size 5000
+  --config-path /config.json
 ```
 
-**NOTE**: There are multiple optional arguments on the `--chain-config` parameter. For a full explanation of the optional arguments, use the `--help` flag:
-
-  ```bash
-  docker run --rm -it \
-    ghcr.io/cowprotocol/watch-tower:latest \
-    run \
-    --help
-  ```
+**NOTE**: See the example `config.json.example` for an example configuration file.
 
 ### DAppNode
 
@@ -78,7 +71,7 @@ The watch-tower monitors the following events:
 When a new event is discovered, the watch-tower will:
 
 1. Fetch the conditional order(s) from the blockchain.
-2. Post the **discrete** order(s) to the [CoW Protocol `OrderBook` API](https://api.cow.fi/docs/#/).
+2. Post the **discrete** order(s) to the [CoW Protocol `OrderBook` API](https://docs.cow.fi/cow-protocol/reference/apis/orderbook).
 
 ### Storage (registry)
 
@@ -186,7 +179,7 @@ It is recommended to test against the Goerli testnet. To run the watch-tower:
 # Install dependencies
 yarn
 # Run watch-tower
-yarn cli run --chain-config <rpc>,<deployment-block> --page-size 5000
+yarn cli run --config-path ./config.json
 ```
 
 ### Testing

config.json.example

@@ -0,0 +1,12 @@
+{
+    "networks": [
+        {
+            "name": "mainnet",
+            "rpc": "ws://172.20.0.5:8546",
+            "deploymentBlock": 17883049,
+            "filterPolicy": {
+                "defaultAction": "ACCEPT"
+            }
+        }
+    ]
+}

package.json

@@ -14,13 +14,15 @@
     "lint:fix": "eslint --fix \"src/**/*.ts\" && prettier --write \"src/**/*.ts\"",
     "test": "yarn build && jest ./dist",
     "typechain": "typechain --target ethers-v5 --out-dir src/types/generated/ \"abi/*.json\"",
-    "prepare": "husky install && yarn typechain",
+    "configschema": "node ./src/types/build.js",
+    "prepare": "husky install && yarn typechain && yarn configschema",
     "cli": "ts-node src/index.ts"
   },
   "devDependencies": {
     "@commitlint/cli": "^17.6.7",
     "@commitlint/config-conventional": "^17.6.7",
     "@typechain/ethers-v5": "^11.1.1",
+    "@types/ajv": "^1.0.0",
     "@types/express": "^4.17.18",
     "@types/jest": "^29.5.3",
     "@types/node": "^18.16.3",
@@ -35,6 +37,7 @@
     "eslint-plugin-unused-imports": "^3.0.0",
     "husky": "^8.0.3",
     "jest": "^28.1.3",
+    "json-schema-to-typescript": "^13.1.2",
     "lint-staged": "^14.0.1",
     "prettier": "^2.8.8",
     "ts-node": "^10.9.1",
@@ -44,6 +47,8 @@
     "@commander-js/extra-typings": "^11.0.0",
     "@cowprotocol/contracts": "^1.4.0",
     "@cowprotocol/cow-sdk": "^4.0.3",
+    "ajv": "^8.12.0",
+    "ajv-formats": "^2.1.1",
     "chalk": "^4.1.2",
     "ethers": "^5.7.2",
     "express": "^4.18.2",

src/commands/index.ts

@@ -1,5 +1,2 @@
 export * from "./run";
-export * from "./runMulti";
-export * from "./replayBlock";
-export * from "./replayTx";
 export * from "./dumpDb";

src/commands/run.ts

@@ -1,4 +1,4 @@
-import { RunSingleOptions } from "../types";
+import { RunOptions } from "../types";
 import { getLogger, DBService } from "../utils";
 import { ChainContext } from "../domain";
 import { ApiService } from "../utils/api";
@@ -7,9 +7,9 @@ import { ApiService } from "../utils/api";
  * Run the watch-tower 👀🐮
  * @param options Specified by the CLI / environment for running the watch-tower
  */
-export async function run(options: RunSingleOptions) {
+export async function run(options: RunOptions) {
   const log = getLogger("commands:run");
-  const { oneShot, disableApi, apiPort, databasePath } = options;
+  const { oneShot, disableApi, apiPort, databasePath, networks } = options;
 
   // Open the database
   const storage = DBService.getInstance(databasePath);
@@ -33,11 +33,27 @@ export async function run(options: RunSingleOptions) {
 
   let exitCode = 0;
   try {
-    const chainContext = await ChainContext.init(options, storage);
-    const runPromise = chainContext.warmUp(oneShot);
+    const chainContexts = await Promise.all(
+      networks.map((network) => {
+        const { name } = network;
+        log.info(`Starting chain ${name}...`);
+        return ChainContext.init(
+          {
+            ...options,
+            ...network,
+          },
+          storage
+        );
+      })
+    );
 
-    // Run the block watcher after warm up for the chain
-    await runPromise;
+    // Run the block watcher after warm up for each chain
+    const runPromises = chainContexts.map(async (context) => {
+      return context.warmUp(oneShot);
+    });
+
+    // Run all the chain contexts
+    await Promise.all(runPromises);
   } catch (error) {
     log.error("Unexpected error thrown when running watchtower", error);
     exitCode = 1;

src/domain/chainContext.ts

@@ -1,5 +1,4 @@
 import {
-  RunSingleOptions,
   Registry,
   ReplayPlan,
   ConditionalOrderCreatedEvent,
@@ -8,6 +7,7 @@ import {
   Multicall3__factory,
   RegistryBlock,
   blockToRegistryBlock,
+  ContextOptions,
 } from "../types";
 import {
   SupportedChainId,
@@ -34,10 +34,11 @@ import {
 import { hexZeroPad } from "ethers/lib/utils";
 import { FilterPolicy } from "../utils/filterPolicy";
 
-const WATCHDOG_FREQUENCY = 5 * 1000; // 5 seconds
+const WATCHDOG_FREQUENCY_SECS = 5; // 5 seconds
+const WATCHDOG_TIMEOUT_DEFAULT_SECS = 30;
 
 const MULTICALL3 = "0xcA11bde05977b3631167028862bE2a173976CA11";
-const FILTER_FREQUENCY_SECS = 60 * 60; // 1 hour
+const PAGE_SIZE_DEFAULT = 5000;
 
 export const SDK_BACKOFF_NUM_OF_ATTEMPTS = 5;
 
@@ -97,7 +98,7 @@ export class ChainContext {
   multicall: Multicall3;
 
   protected constructor(
-    options: RunSingleOptions,
+    options: ContextOptions,
     provider: providers.Provider,
     chainId: SupportedChainId,
     registry: Registry
@@ -109,12 +110,12 @@ export class ChainContext {
       watchdogTimeout,
       owners,
       orderBookApi,
-      filterPolicyConfig,
+      filterPolicy,
     } = options;
     this.deploymentBlock = deploymentBlock;
-    this.pageSize = pageSize;
+    this.pageSize = pageSize ?? PAGE_SIZE_DEFAULT;
     this.dryRun = dryRun;
-    this.watchdogTimeout = watchdogTimeout;
+    this.watchdogTimeout = watchdogTimeout ?? WATCHDOG_TIMEOUT_DEFAULT_SECS;
     this.addresses = owners;
 
     this.provider = provider;
@@ -135,12 +136,7 @@ export class ChainContext {
       },
     });
 
-    this.filterPolicy = filterPolicyConfig
-      ? new FilterPolicy({
-          configBaseUrl: filterPolicyConfig,
-          // configAuthToken: filterPolicyConfigAuthToken, // TODO: Implement authToken
-        })
-      : undefined;
+    this.filterPolicy = new FilterPolicy(filterPolicy);
     this.contract = composableCowContract(this.provider, this.chainId);
     this.multicall = Multicall3__factory.connect(MULTICALL3, this.provider);
   }
@@ -153,7 +149,7 @@ export class ChainContext {
    * @returns A chain context that is monitoring for orders on the chain.
    */
   public static async init(
-    options: RunSingleOptions,
+    options: ContextOptions,
     storage: DBService
   ): Promise<ChainContext> {
     const { rpc, deploymentBlock } = options;
@@ -384,7 +380,7 @@ export class ChainContext {
     // pod, we don't exit, but we do log an error and set the sync status to unknown.
     while (true) {
       // sleep for 5 seconds
-      await asyncSleep(WATCHDOG_FREQUENCY);
+      await asyncSleep(WATCHDOG_FREQUENCY_SECS * 1000);
       const now = Math.floor(new Date().getTime() / 1000);
       const timeElapsed = now - lastBlockReceived.timestamp;
 
@@ -460,7 +456,7 @@ async function processBlock(
   blockNumberOverride?: number,
   blockTimestampOverride?: number
 ) {
-  const { provider, chainId, filterPolicy } = context;
+  const { provider, chainId } = context;
   const timer = processBlockDurationSeconds
     .labels(context.chainId.toString())
     .startTimer();
@@ -470,21 +466,6 @@ async function processBlock(
     block.number.toString()
   );
 
-  // Refresh the policy every hour
-  // NOTE: This is a temporary solution until we have a better way to update the filter policy
-  const blocksPerFilterFrequency =
-    FILTER_FREQUENCY_SECS /
-    (context.chainId === SupportedChainId.GNOSIS_CHAIN ? 5 : 12); // 5 seconds for gnosis, 12 seconds for mainnet
-  if (
-    filterPolicy &&
-    block.number % (FILTER_FREQUENCY_SECS / blocksPerFilterFrequency) == 0
-  ) {
-    filterPolicy.reloadPolicies().catch((error) => {
-      console.log(`Error fetching the filter policy config for chain `, error);
-      return null;
-    });
-  }
-
   // Transaction watcher for adding new contracts
   let hasErrors = false;
   for (const event of events) {