[Improvement/Bug?] Node will never start when Snapshot.StartHeight is set statically and NetworkHistory.Initialise.MinimumBlockCount is big.

[daniel1302]

Problem encountered

Let's say a person who wants to start data from a Remote Snapshot sets Snapshot.StartHeight statically to X, and he wants to have 100k+ blocks in the data node. Usually core will never start.

The Tendermint limits vega to start only from the 10 last remote snapshots- At the moment it is 3000 blocks (300 blocks * 10 snapshots).

3000 blocks is about 30 min of the network life. Syncing 100k blocks from the network history may take longer than 30 minutes.

So the workflow is

1. Sync data-node - let's say it takes 40 min
2. Start core from remote snapshot - after 40 min the statically selected snapshot is not available anymore from the tendermint...

In the above case people are getting:

core.protocol.snapshot        snapshot/engine.go:322        The block height of the received snapshot does not match the expected one, rejecting offer        {"snapshot-height": 26692300, "expected-height": 26531800}

Observed behaviour

People are not able to start their node from a remote snapshot under some conditions.

Expected behaviour

People should be able to start.

1. We probably should WARN them.
2. Or We can sync the core on the beginning start of the data node then let the data node replay, and then replay missing blocks?

At the moment second option looks more reasonable as a person who sets Snapshot.StartHeight to static block wishes to start from that block for some reason.

Steps to reproduce

1. Set the `Snapshot.StartHeight` to the last available snapshot in the vega-core config
2. Set the `NetworkHistory.Initialise.MinimumBlockCount` to 500000 in the data node config
3. Start data-node
4. Wait until it fails

Software version

any

Failing test

No response

Jenkins run

No response

Configuration used

No response

Relevant log output

No response