archive mode doesn't recognize missed blocks

[conorsch]

When running penumbra-reindexer archive, the program will pick up from the most recent block height. That's a sane default. However, it's possible that the sqlite db could be missing blocks, due to operator error. In that case, repeated runs of penumbra-reindexer archive will not backfill missing blocks, even if the local node state knows about them. Thereafter, running penumbra-reindexer regen will fail when encountering a missing block:

2025-01-23T11:51:09.047656Z  INFO init_then_run_to{genesis_height=501975 version=V0o80 first_block=501975 last_block=Some(2611800)}: penumbra_reindexer::penumbra: reached height 2609600
2025-01-23T11:51:09.724047Z  WARN init_then_run_to{genesis_height=501975 version=V0o80 first_block=501975 last_block=Some(2611800)}:end_block{height=2609640}:shielded_pool: penumbra_shielded_pool::fmd: decreasing clue count at height 2609640: 31 then 17
2025-01-23T11:51:10.243356Z  INFO cnidarium::storage: dispatcher task has terminated
Error: missing block at height 2609671

When I encountered that error, I cross-checked the reindexer sqlite3 db, and was able to identify the gap:

$ sqlite3 reindexer_archive.bin
SQLite version 3.40.1 2022-12-28 14:03:47
Enter ".help" for usage hints.
sqlite> WITH numbered_blocks AS (
    SELECT height,
           LEAD(height) OVER (ORDER BY height) as next_height
    FROM blocks
    WHERE height <= 3000000
)
SELECT height + 1 as gap_start, next_height - 1 as gap_end
FROM numbered_blocks
WHERE next_height - height > 1;
2609671|2611799

As a workaround, I edited the source code here to return Ok(Some((2609671, 2611799))); and that resolved the issue.

Making the reindexer automatically heal these gaps would be a significant lift. However, it might be worthwhile to run the check-for-gaps query and emit a warning, or error, if any are found.

Mostly I'm filing this issue for posterity, so future operators have a reference to recover if they encounter this scenario. Encountered this while working on #21.