Bad UX when activating blobs archive (bug?)

[wmitsuda]

Today's main, I just enabled blobs archive in an existing mainnet node with --caplin.blobs-archive.

At first it seemed it was going to download +300GB according to the logs:

[INFO] [02-07|20:18:29.226] [Gossip] Subscribed to topic             topic=/eth2/6a95a1a9/beacon_block/ssz_snappy
[INFO] [02-07|20:18:29.226] [Gossip] Subscribed to topic             topic=/eth2/6a95a1a9/blob_sidecar_8/ssz_snappy
[INFO] [02-07|20:18:40.191] [1/6 OtterSync] Downloading              progress="(1421/1421 files) 86.75% - 1.7TB/2.0TB" time-left=999hrs:99m total-time=40s download-rate=0B/s completion-rate=0B/s alloc=12.4GB sys
=12.9GB
[INFO] [02-07|20:18:40.235] [snapshots] no progress yet              files=186 list=v1-009000-009100-beaconblocks.seg,v1-009860-009870-blobsidecars.seg,v1-004400-004500-beaconblocks.seg,v1-009850-009860-beaconbl
ocks.seg,v1-010710-010720-beaconblocks.seg,...
[INFO] [02-07|20:19:00.191] [1/6 OtterSync] Downloading              progress="(1421/1421 files) 86.84% - 1.7TB/2.0TB" time-left=0hrs:47m total-time=1m0s download-rate=94.3MB/s completion-rate=94.3MB/s alloc=12.
8GB sys=13.2GB
[INFO] [02-07|20:19:00.278] [snapshots] no progress yet              files=173 list=v1-010470-010480-blobsidecars.seg,v1-010690-010700-beaconblocks.seg,v1-010400-010410-beaconblocks.seg,v1-008890-008900-blobside
cars.seg,v1-010670-010680-blobsidecars.seg,...
[INFO] [02-07|20:19:20.191] [1/6 OtterSync] Downloading              progress="(1421/1421 files) 86.94% - 1.7TB/2.0TB" time-left=0hrs:41m total-time=1m20s download-rate=108.3MB/s completion-rate=107.9MB/s alloc=
13.3GB sys=13.7GB

and that's fine, that's something expected given the flag was explicitly added. However after finishing that, it added more +300GB of downloads, see the next logs:

[INFO] [02-07|21:08:20.256] [1/6 OtterSync] Downloading              progress="(1421/1637 files) 100.00% - 2.0TB/2.0TB" time-left=0hrs:0m total-time=50m20s download-rate=51.2KB/s completion-rate=51.2KB/s alloc=1
2.5GB sys=18.9GB
[INFO] [02-07|21:08:20.278] [snapshots] no metadata yet              files=216 list=v1-005900-006000-beaconblocks.seg,v1-006300-006400-beaconblocks.seg,v1-004100-004200-beaconblocks.seg,v1-010140-010150-blobside
cars.seg,v1-010430-010440-blobsidecars.seg,...
[INFO] [02-07|21:08:28.218] P2P                                      app=caplin peers=4
[INFO] [02-07|21:08:40.192] [1/6 OtterSync] Downloading              progress="(1421/1637 files) 100.00% - 2.0TB/2.0TB" time-left=0hrs:0m total-time=50m40s download-rate=12.8KB/s completion-rate=12.8KB/s alloc=1
5.1GB sys=18.9GB
[INFO] [02-07|21:08:40.234] [snapshots] no progress yet              files=181 list=v1-009110-009120-blobsidecars.seg,v1-009080-009090-blobsidecars.seg,v1-010310-010320-beaconblocks.seg,v1-009970-009980-blobside
cars.seg,v1-010300-010310-beaconblocks.seg,...
[INFO] [02-07|21:09:00.191] [1/6 OtterSync] Downloading              progress="(1637/1637 files) 85.10% - 2.0TB/2.3TB" time-left=19hrs:9m total-time=51m0s download-rate=5.2MB/s completion-rate=5.0MB/s alloc=15.5
GB sys=18.9GB
[INFO] [02-07|21:09:00.192] [p2p] GoodPeers                          eth68=3
[INFO] [02-07|21:09:00.255] [txpool] stat                            pending=0 baseFee=0 queued=0 alloc=15.5GB sys=18.9GB
[INFO] [02-07|21:09:00.268] [snapshots] no progress yet              files=8 list=v1-010420-010430-blobsidecars.seg,v1-010730-010740-blobsidecars.seg,v1-010720-010730-beaconblocks.seg,v1-010460-010470-blobsideca
rs.seg,v1-009550-009560-blobsidecars.seg,...
[INFO] [02-07|21:09:01.017] [mem] memory stats                       Rss=21.7GB Size=0B Pss=21.7GB SharedClean=2.7MB SharedDirty=0B PrivateClean=7.8GB PrivateDirty=13.8GB Referenced=11.1GB Anonymous=12.9GB Swap=
4.8GB alloc=15.5GB sys=18.9GB
[INFO] [02-07|21:09:20.207] [1/6 OtterSync] Downloading              progress="(1637/1637 files) 85.19% - 2.0TB/2.3TB" time-left=0hrs:57m total-time=51m20s download-rate=103.3MB/s completion-rate=103.3MB/s alloc
=15.8GB sys=18.9GB
[INFO] [02-07|21:09:21.146] [snapshots] no progress yet              files=5 list=v1-010460-010470-blobsidecars.seg,v1-010400-010410-blobsidecars.seg,v1-010680-010690-blobsidecars.seg,v1-010420-010430-blobsideca
rs.seg,v1-010730-010740-blobsidecars.seg
[INFO] [02-07|21:09:28.218] P2P                                      app=caplin peers=0
[INFO] [02-07|21:09:40.191] [1/6 OtterSync] Downloading              progress="(1637/1637 files) 85.25% - 2.0TB/2.3TB" time-left=1hrs:13m total-time=51m40s download-rate=80.0MB/s completion-rate=80.1MB/s alloc=1
6.1GB sys=18.9GB

Not sure if there is a bug on accounting of initial added download or it is expected that those files are added in batches, in this case it seems at least UX could be improved because we are surprising the user: "oh right, it'll download +300GB... good, it finished, oh wait it is going to download +300GB, how much is the total?"