Fast Recovery From App Hash Mismatch

Context

cronosd rollback provide a way to rollback and recover from app-hash mismatch failure, but it takes hours to run, here is a faster way to do it.

Prerequisite

• tendermint v0.34.22 cli, installed with:

  # rocksdb
  $ go install -tags rocksdb github.com/tendermint/tendermint/cmd/[email protected]
  # leveldb
  $ go install github.com/tendermint/tendermint/cmd/[email protected]
  

• python-iavl, could be run with nix on the fly:

  $ # rocksdb
  $ nix run github:yihuang/python-iavl/a1aeedf
  $ # leveldb
  $ nix run github:yihuang/python-iavl/a1aeedf#iavl-cli-leveldb
  

Fast Rollback

1. Disable iavl fastnode

   fastnode reindex alone is slow enough, disable it to make it faster to start up.

   Set iavl-disable-fastnode = true in app.toml

2. Rollback tendermint state

   $ tendermint rollback --home /chain/.cronosd/
   Rolled back state to height $VER and hash [...]
   

3. Fast rollback app state, replace $VER to the version number shown in the above command.

   $ nix run github:yihuang/python-iavl/a1aeedf -- fast-rollback --db /chain/.cronosd/data/application.db --target $VER