Fast Recovery From App Hash Mismatch

Context

cronosd rollback provide a way to rollback and recover from app-hash mismatch failure, but it takes hours to run, here is a faster way to do it.

Prerequisite

• tendermint v0.34.22 cli, installed with:

  # rocksdb
  $ go install -tags rocksdb github.com/tendermint/tendermint/cmd/[email protected]
  # leveldb
  $ go install github.com/tendermint/tendermint/cmd/[email protected]
  

• python-iavl, could be run with nix on the fly:

  $ # rocksdb
  $ nix run github:yihuang/python-iavl/a1aeedf
  $ # leveldb
  $ nix run github:yihuang/python-iavl/a1aeedf#iavl-cli-rocksdb
  

Fast Rollback

NOTICE: python-iavl only support rocksdb for now.

1. Disable iavl fastnode fastnode reindex itself is slow enough, to make it faster, disable it. Set iavl-disable-fastnode = true in app.toml
2. Rollback tendermint state

   $ tendermint rollback --home /chain/.cronosd/
   Rolled back state to height $VER and hash [...]
   

3. Fast rollback app state, replace $VER to the version number shown in the above command.

   $ nix run github:yihuang/python-iavl/a1aeedf -- fast-rollback --db /path/to/data/application.db --target $VER