[Support Ticket]: boostd v2.1.1, LevelDB, retrieval errors (multihash not found)

[Shekelme]

Boost component

• boost daemon - storage providers
• boost client
• boost UI
• boost data-transfer
• boost index-provider
• booster-http
• booster-bitswap
• LID Database - Yugabyte/LevelDB
• boostd-data
• Other

Boost Version

boostd version 2.1.1+mainnet+git.e9d18ac

Describe the problem

After the boost migration from version 1.7.5 to version 2.1.1 (using LevelDB since this is a small miner), all retrievals end with an error of this type:

In some more details:

Retrieval 1701681219006044710
CreatedAt	2023-12-06 21:52:13.795 (41m ago)
Client Peer ID	12D3KooWKN5awGjrcSvcXZzHSk1h2zV4fageVjsYuCrZUNzyDkGp
Transfer ID	1701681219006044710
Retrieval Deal ID	1701681216229037819
Deal Data Root CID	bafkreidsq22vequ5dfwlmrmxlztqurqdxuyrupnoo6npyiqdbt5mps3nu4
Piece CID	
Price per byte	0 atto
Unseal price	0 atto
Payment Interval	0 B (0 bytes)
Payment Interval Increase	0 B (0 bytes)
Sent	0 B (0 bytes)
Status	Errored
Event Logs
2023-12-06 21:52:13.789		DT:Open		unpaid retrieval
2023-12-06 21:52:13.789	0ms	DT:TransferRequestQueued		
2023-12-06 21:52:13.792	3ms	DT:Error		getting pieces for cid bafkreidsq22vequ5dfwlmrmxlztqurqdxuyrupnoo6npyiqdbt5mps3nu4: multihash 12207286b552429d196cb645975e670a4603bd311a3dae779afc22030cfac7cb6da7: not found
2023-12-06 21:52:13.797	5ms	Open	New	
2023-12-06 21:52:13.805	8ms	DataTransferError	Errored

At first I thought the error was that the boostd-data service was not running at the time of the pieceinfo migration.
Then I found that this command itself was wrong. But after correcting the command, despite the fact that the migration log seems to report the successful completion of the pieceinfo migration, still only three indexed deals received and sealed after the upgrade are displayed on the lid page.

The migration was carried out using the following commands:
migrate-lid leveldb dagstore
The migration was not completed on the first attempt, I had to run the migration several times until all migrations were gradually completed (judging by the messages that the command returned upon completion), except for one (timestamp 2023-12-05T00:03:52.234 in the log).

Then my erroneous command looked like migrate-lid leveldb dagstore pieceinfo, so I assume that the pieceinfo migration has not been carried out. But since I was still informed about one unfinished migration, I repeated the command a couple of times in the hope that the migration would still be completed. Timestamps 2023-12-05T16:37:46.938 to 2023-12-05T17:02:41.372.

Today (2023-12-06T17:26:46.424) I tried to migrate pieceinfo with the boostd-data service running, but again with the wrong command.
Then finally with correct command migrate-lid leveldb pieceinfo (timestamp 2023-12-06T17:47:53.396) - 621 piece infos migrated, no errors, but still no any single pre-upgrade deal on the Local Index Directory page.
I tried to repeat the pieceinfo migration even with the boostd-data service turned off, no attempts gave any result (except for a negative result).
migrate-leveldb.log

Logging Information

Attached (too long to paste into the body)

Repo Steps

1. Run '...'
2. Do '...'
3. See error '...'
   ...

[LexLuthr]

@Shekelme If you still have the older dagstore safe then I would recommend rerunning the migration. I highly recommend YugabyteDB instead of levelDB for performance and easy of use. Once you are done migrating, we can recover the 3 new deals using the recovery tools. Please make sure to follow the procedure correctly this time.

[Shekelme]

I still remain confident that for a small miner of several tens of terabytes of raw power, there is no need to raise a cluster of at least three YugabyteDB instances.
Can we extend 1m timeout for migrate-lid leveldb dagstore command?

[LexLuthr]

Multiple SPs are running a single node Yugabyte cluster for smaller miners. The timeout is defined within the code and is not part of the execution command. I would not recommend increasing it either. 60 seconds is enough time for a shard to be read and written back unless something is blocking the execution.

[Shekelme]

I don't get it. I deleted the previously generated .boost/lid folder to make a new migration from scratch. Of course, boostd-data and boostd has been offline (I even disabled the boostd-data service).
I wonder if the picture will change if I try to migrate to YugabyteDB.

[LexLuthr]

@Shekelme I think there is something preventing your system from migrating properly or it might be migrating the pieceInfo in a different directory. Have you checked if there are any other LID directories maybe? With Yugabyte, you have more control and we can always access the DB to check what might be the issue.

[Shekelme]

No, I think I found the reason.....
migrate-lid leveldb creates the lid folder at $BOOST_PATH, while boostd-data by default use empty folder at ~/.boost

[LexLuthr]

That explains the problems. Both commands have an option to set the directory so start boostd-data service with the same path as boostd.

[Shekelme]

OK, how can we recover 4 deals sealed after upgrade?

[LexLuthr]

$ boostd recover lid --help
NAME:
   boostd recover lid - lid

USAGE:
   boostd recover lid [command options] [arguments...]

OPTIONS:
   --api-fullnode value           the endpoint for the full node API
   --api-storage value            the endpoint for the storage node API
   --api-lid value                the endpoint for the LID API
   --recover-dir value            location to store progress of recovery (default: "~/.boost-recover")
   --repo value                   location to boost repo (default: "~/.boost")
   --sector-id value              sector-id (default: 0)
   --add-index-throttle value     (default: 4)
   --add-index-concurrency value  the maximum number of parallel tasks that a single add index operation can be split into (default: 8)
   --ignore-commp                 whether we should ignore sanity check of local data vs chain data (default: false)
   --ignore-lid                   whether we should ignore lid (default: false)
   --help, -h                     show help

Please make sure to shutdown boostd for recovery but do not shutdown boostd-data as it is required by the recovery service to insert the data in LID. If you know the sector IDs of the missing deals then please use --sector-id flag and it will be a fast recovery. The sector IDs can be found from the deal details page.