chore: cleanup repo, readme, add cli script (#109)

# Description
This PR finishes off feedback from #60.

# Changes

- [x] Comprehensive editorial review and adjustments to `README.md` to
reflect current state of the repo
- [x] Long-awaited addition of `cli` script into `package.json`
- [x] Parameterized the database location (configured via optional `CLI`
option)
- [x] Remove all references to Tenderly

## How to test

1. Use latest `pr` `docker` image.
2. Follow instructions in `README.md` for running from a `docker` image.
3. Confirm that the respective chain achieves sync.

## Related Issues

Fixes #60

.github/ISSUE_TEMPLATE/bug_report.md

@@ -26,8 +26,8 @@ A clear and concise description of what you expected to happen.
 ### Screenshots/logs
 If applicable, add screenshots or logs to help explain your problem.
 
-### Tenderly watch-tower version/commit hash
-State the version of tenderly-watch-tower where you've encountered the bug or, if built off a specific commit, the relevant commit hash.
+### watch-tower version/commit hash
+State the version of watch-tower where you've encountered the bug or, if built off a specific commit, the relevant commit hash.
 - e.g. `v0.9` or `ed53bcd`
 
 ### Additional context

README.md

@@ -1,78 +1,107 @@
-# Watch Tower for Composable CoWs 🐮🎶
+# Watch-Tower for Composable CoWs 🐮🎶
 
-A watch tower has been implementing using standalone ethers.js. By means of _emitted Event_ and new block monitoring, conditional orders can run autonomously.
+## Overview
 
-Notably, with the `ConditionalOrderCreated` and `MerkleRootSet` events, multiple conditional orders can be created for one safe - in doing so, the actions maintain a registry of:
+The [`ComposableCoW`](https://github.com/cowprotocol/composable-cow) conditional order framework requires a watch-tower to monitor the blockchain for new orders, and to post them to the [CoW Protocol `OrderBook` API](https://api.cow.fi/docs/#/). The watch-tower is a standalone application that can be run locally as a script for development, or deployed as a docker container to a server, or dappnode.
 
-1. Safes that have created _at least one conditional order_.
-2. All payloads for conditional orders by safe that have not expired or been cancelled.
-3. All part orders by `orderUid` containing their status (`SUBMITTED`, `FILLED`) - the `Trade` on `GPv2Settlement` is monitored to determine if an order is `FILLED`.
+## Deployment
 
-As orders expire, or are cancelled, they are removed from the registry to conserve storage space.
+If running your own watch-tower instance, you will need the following:
+
+- An RPC node connected to the Ethereum mainnet, Gnosis Chain, or Goerli.
+- Internet access to the [CoW Protocol `OrderBook` API](https://api.cow.fi/docs/#/).
+
+**CAUTION**: Conditional order types may consume considerable RPC calls.
+
+**NOTE**: `deployment-block` refers to the block number at which the **`ComposableCoW`** contract was deployed to the respective chain. This is used to optimise the watch-tower by only fetching events from the blockchain after this block number. Refer to [Deployed Contracts](https://github.com/cowprotocol/composable-cow#deployed-contracts) for the respective chains.
+
+**NOTE**: The `--page-size` option is used to specify the number of blocks to fetch from the blockchain when querying historical events (`eth_getLogs`). The default is `5000`, which is the maximum number of blocks that can be fetched in a single request from Infura. If you are running the watch-tower against your own RPC, you may want to set this to `0` to fetch all blocks in one request, as opposed to paging requests.
 
-### Local testing
 
-This is assuming that you have followed the instructions for deploying the stack on `anvil` in [local deployment](#Local-deployment)
+### Docker
 
-From the root directory of the repository:
+The preferred method of deployment is using `docker`. The watch-tower is available as a docker image on [GitHub](https://github.com/cowprotocol/watch-tower/pkgs/container/watch-tower). The tags available are:
+
+- `latest` - the latest version of the watch-tower.
+- `vX.Y.Z` - the version of the watch-tower.
+- `main` - the latest version of the watch-tower on the `main` branch.
+- `pr-<PR_NUMBER>` - the latest version of the watch-tower on the PR.
+
+As an example, to run the latest version of the watch-tower via `docker`:
 
 ```bash
-yarn
-yarn ts-node ./src/index.ts run --rpc http://127.0.0.1:8545 --deployment-block <deployment-block> --contract-address <contract-address> --page-size 0
+docker run --rm -it \
+  ghcr.io/cowprotocol/watch-tower:latest \
+  run \
+  --rpc <rpc-url> \
+  --deployment-block <deployment-block> \
+  --page-size 5000
 ```
 
-### Watch tower state
+### Dappnode
 
-The watch tower stores data in a leveldb database. The database is stored by default in the `./database` directory. Writes to the database are batched, and if writes fail, the watch tower will throw an error and exit. On restarting, the watch tower will attempt to re-process the last block that was indexed.
+**TODO**: Add instructions for deploying to Dappnode.
 
-### Deployment
+### Running locally
 
-If running your own watch tower, or deploying for production:
+#### Requirements
+
+- `node` (`>= v16.18.0`)
+- `yarn`
+
+#### CLI
 
 ```bash
+# Install dependencies
 yarn
-yarn ts-node ./src/index.ts run --rpc <rpc-url> --deployment-block <deployment-block> --contract-address <contract-address> --page-size 0
+# Run watch-tower
+yarn cli run --rpc <rpc-url> --deployment-block <deployment-block> --page-size 5000
 ```
 
-## Developers
+## Architecture
 
-### Requirements
+### Events
 
-- `node` (`>= v16.18.0`)
-- `yarn`
-- `npm`
+The watch-tower monitors the following events:
 
-### Environment setup
+- `ConditionalOrderCreated` - emitted when a _single_ new conditional order is created.
+- `MerkleRootSet` - emitted when a new merkle root (ie. `n` conditional orders) is set for a safe.
 
-Copy the `.env.example` to `.env` and set the applicable configuration variables for the testing / deployment environment.
+When a new event is discovered, the watch-tower will:
 
-#### Local deployment
+1. Fetch the conditional order(s) from the blockchain.
+2. Post the **discrete** order(s) to the [CoW Protocol `OrderBook` API](https://api.cow.fi/docs/#/).
 
-For local integration testing, such as local debugging of tenderly actions, it may be useful deploying to a _forked_ mainnet environment. Information on how to do this is available in the [ComposableCoW repository](https://github.com/cowprotocol/composable-cow).
+### Storage (registry)
 
-#### Run the standalone watch tower
+The watch-tower stores the following state:
 
-```bash
-yarn
-yarn ts-node ./src/index.ts run --rpc <rpc-url> --deployment-block <deployment-block> --page-size 0
-```
+- All owners (ie. safes) that have created at least one conditional order.
+- All conditional orders by safe that have not expired or been cancelled.
 
-> Useful for debugging locally the actions. Also could be used to create an order for an old block in case there was a failure of WatchTowers indexing it.
+As orders expire, or are cancelled, they are removed from the registry to conserve storage space.
 
-```bash
-# Install dependencies
-yarn
+#### Database
 
-# Run actions locally
-#   - It will synchronize the database with the blockchain from the deployment block
-#   - It will start watching and processing new blocks
-#   - As a result, new Composable Cow orders will be discovered and posted to the OrderBook API
-yarn ts-node ./src/index.ts run --rpc <rpc-url> --deployment-block <deployment-block> --page-size 0
-```
+The chosen architecture for the storage is a NoSQL (key-value) store. The watch-tower uses the following:
+
+- [`level`](https://www.npmjs.com/package/level)
+- Default location: `$PWD/database`
+
+`LevelDB` is chosen it it provides [ACID](https://en.wikipedia.org/wiki/ACID) guarantees, and is a simple key-value store. The watch-tower uses the `level` package to provide a simple interface to the database. All writes are batched, and if a write fails, the watch-tower will throw an error and exit. On restarting, the watch-tower will attempt to re-process from the last block that was successfully indexed, resulting in the database becoming eventually consistent with the blockchain.
+
+##### Schema
 
-## Logging
+The following keys are used:
 
-To control logging level, you can set the `LOG_LEVEL` environment variable with one of the following values: `TRACE`, `DEBUG`, `INFO`, `WARN`, `ERROR`, `SILENT`:
+- `LAST_PROCESSED_BLOCK` - the last block (number, timestamp, and hash) that was processed by the watch-tower.
+- `CONDITIONAL_ORDER_REGISTRY` - the registry of conditional orders by safe.
+- `CONDITIONAL_ORDER_REGISTRY_VERSION` - the version of the registry. This is used to migrate the registry when the schema changes.
+- `LAST_NOTIFIED_ERROR` - the last time an error was notified via Slack. This is used to prevent spamming the slack channel.
+
+### Logging
+
+To control logging level, you can set the `LOG_LEVEL` environment variable with one of the following values: `TRACE`, `DEBUG`, `INFO`, `WARN`, `ERROR`:
 
 ```ini
 LOG_LEVEL=WARN
@@ -115,12 +144,12 @@ LOG_LEVEL=chainContext:processBlock:(100|1):(\d*)$=DEBUG
 Combine all of the above to control the log level of any modules:
 
 ```ini
- LOG_LEVEL="WARN,commands=DEBUG,^checkForAndPlaceOrder=WARN,^chainContext=INFO,_checkForAndPlaceOrder:1:=INFO" yarn ts-node ./src/index.ts
+ LOG_LEVEL="WARN,commands=DEBUG,^checkForAndPlaceOrder=WARN,^chainContext=INFO,_checkForAndPlaceOrder:1:=INFO" yarn cli
 ```
 
-## API Server
+### API Server
 
-The run command will expose by default a server on port `8080`.
+Commands that run the watch-tower in a watching mode, will also start an API server. By default the API server will start on port `8080`. You can change the port using the `--api-port <apiPort>` CLI option.
 
 The server exposes automatically:
 
@@ -129,8 +158,58 @@ The server exposes automatically:
   - Dump Database: `http://localhost:8080/api/dump/:chainId` e.g. [http://localhost:8080/api/dump/1](http://localhost:8080/api/dump/1)
 - Prometheus Metrics: [http://localhost:8080/metrics](http://localhost:8080/metrics)
 
-You can prevent the server from starting by setting the `--disable-api` flag for the `run` command.
+You can prevent the API server from starting by setting the `--disable-api` flag for the `run` command.
 
-Additionally, you can change the default port using the `--api-port <apiPort>` CLI option.
+The `/api/version` endpoint, exposes the information in the `package.json`. This can be helpful to identify the version of the watch-tower. Additionally, for environments using `docker`, the environment variable `DOCKER_IMAGE_TAG` can be used to specify the Docker image tag used.
+
+## Developers
 
-The `/api/version` endpoint, exposes the information in the package.json. This can be helpful to identify the version of the watch tower. Additionally, for environments using docker, the environment variable `DOCKER_IMAGE_TAG` can be used to specify the Docker image tag used.
+### Requirements
+
+- `node` (`>= v16.18.0`)
+- `yarn`
+- `npm`
+
+### Local development
+
+It is recommended to test against the Goerli testnet. To run the watch-tower:
+
+```bash
+# Install dependencies
+yarn
+# Run watch-tower
+yarn cli run --rpc <rpc-url> --deployment-block <deployment-block> --page-size 5000
+```
+
+### Testing
+
+To run the tests:
+
+```bash
+yarn test
+```
+
+### Linting / Formatting
+
+```bash
+# To lint the code
+yarn lint
+```
+
+```bash
+# To fix linting errors
+yarn lint:fix
+```
+
+```bash
+# To format the code
+yarn fmt
+```
+
+### Building the docker image
+
+To build the docker image:
+
+```bash
+docker build -t watch-tower .
+```

package.json

@@ -14,7 +14,8 @@
     "lint:fix": "eslint --fix \"src/**/*.ts\" && prettier --write \"src/**/*.ts\"",
     "test": "yarn build && jest ./dist",
     "typechain": "typechain --target ethers-v5 --out-dir src/types/generated/ \"abi/*.json\"",
-    "prepare": "husky install && yarn typechain"
+    "prepare": "husky install && yarn typechain",
+    "cli": "ts-node src/index.ts"
   },
   "devDependencies": {
     "@commitlint/cli": "^17.6.7",

src/commands/dumpDb.ts

@@ -7,9 +7,9 @@ import { DBService, getLogger } from "../utils";
  */
 export async function dumpDb(options: DumpDbOptions) {
   const log = getLogger("commands:dumpDb");
-  const { chainId } = options;
+  const { chainId, databasePath } = options;
 
-  Registry.dump(DBService.getInstance(), chainId.toString())
+  Registry.dump(DBService.getInstance(databasePath), chainId.toString())
     .then((dump) => console.log(dump))
     .catch((error) => {
       log.error("Unexpected thrown when dumping DB", error);

src/commands/run.ts

@@ -15,9 +15,13 @@ export async function run(options: RunOptions) {
     oneShot,
     disableApi,
     apiPort,
+    databasePath,
     watchdogTimeout,
   } = options;
 
+  // Open the database
+  const storage = DBService.getInstance(databasePath);
+
   // Start the API server if it's not disabled
   if (!disableApi) {
     log.info("Starting Rest API server...");
@@ -45,7 +49,7 @@ export async function run(options: RunOptions) {
             rpc,
             deploymentBlock: deploymentBlock[index],
           },
-          DBService.getInstance()
+          storage
         );
       })
     );

src/domain/checkForAndPlaceOrder.ts

@@ -66,7 +66,7 @@ const API_ERRORS_BACKOFF: BackOffApiErrorsDelays = {
 /**
  * Watch for new blocks and check for orders to place
  *
- * @param context tenderly context
+ * @param context chain context
  * @param event block event
  */
 export async function checkForAndPlaceOrder(
@@ -207,7 +207,7 @@ export async function checkForAndPlaceOrder(
   }
 
   // It may be handy in other versions of the watch tower implemented in other languages
-  // (ie. not for Tenderly) to not delete owners, so we can keep track of them.
+  // to not delete owners, so we can keep track of them.
   for (const [owner, conditionalOrders] of Array.from(ownerOrders.entries())) {
     if (conditionalOrders.size === 0) {
       ownerOrders.delete(owner);

src/index.ts

@@ -57,6 +57,11 @@ async function main() {
         .argParser(parseIntOption)
     )
     .addOption(logLevelOption)
+    .option(
+      "--database-path <databasePath>",
+      "Path to the database",
+      "./database"
+    )
     .action((options) => {
       const { logLevel } = options;
       const [pageSize, apiPort, watchdogTimeout] = [
@@ -88,6 +93,11 @@ async function main() {
     .description("Dump database as JSON to STDOUT")
     .requiredOption("--chain-id <chainId>", "Chain ID to dump")
     .addOption(logLevelOption)
+    .option(
+      "--database-path <databasePath>",
+      "Path to the database",
+      "./database"
+    )
     .action((options) => {
       const { logLevel } = options;
       initLogging({ logLevel });
@@ -109,6 +119,11 @@ async function main() {
     .requiredOption("--block <block>", "Block number to replay")
     .option("--dry-run", "Do not publish orders to the OrderBook API", false)
     .addOption(logLevelOption)
+    .option(
+      "--database-path <databasePath>",
+      "Path to the database",
+      "./database"
+    )
     .action((options) => {
       const { logLevel } = options;
       initLogging({ logLevel });
@@ -129,6 +144,11 @@ async function main() {
     .requiredOption("--tx <tx>", "Transaction hash to replay")
     .option("--dry-run", "Do not publish orders to the OrderBook API", false)
     .addOption(logLevelOption)
+    .option(
+      "--database-path <databasePath>",
+      "Path to the database",
+      "./database"
+    )
     .action((options: ReplayTxOptions) => {
       const { logLevel } = options;
       initLogging({ logLevel });

src/types/index.ts

@@ -1,5 +1,6 @@
 export interface LogOptions {
   logLevel: string;
+  databasePath: string;
 }
 
 export interface WatchtowerOptions extends LogOptions {

src/types/model.ts

@@ -126,7 +126,7 @@ export class Registry {
    * Get the raw conditional orders key from the registry.
    * @param storage The storage service to use
    * @param network The network to dump the registry for
-   * @returns The conditional orders that should also be present in the tenderly version
+   * @returns The conditional orders that are in the registry
    */
   public static async dump(
     storage: DBService,