repair index has no effect

[Schweber]

Checking my repository yields the following:

 ~  rustic -P slim -P MEGA check                                                                                                                                                                                                                                                                                       
[INFO] using config /home/sperber/.config/rustic/slim.toml
[INFO] using config /home/sperber/.config/rustic/MEGA.toml
[INFO] repository rclone:MEGA:Rustic: password is correct.
[INFO] using cache at /home/sperber/.cache/rustic/378cce4ff1059d99310953312ff66e49029e0ce98472c6d4190fa10f33458952
[00:00:00] checking Snapshot in cache...  ████████████████████████████████████████  27.38 KiB/27.38 KiB  25.23 MiB/s  (ETA 0s)
[00:00:00] checking Index in cache...     ████████████████████████████████████████  17.67 MiB/17.67 MiB  3.68 GiB/s   (ETA 0s)
[00:00:00] reading index...               ████████████████████████████████████████        128/128[00:00:00] listing packs...               
[WARN] pack 0712bf9f not referenced in index. Can be a parallel backup job. To repair: 'rustic repair index'.
[WARN] pack 0712bf9f not referenced in index. Can be a parallel backup job. To repair: 'rustic repair index'.
[WARN] pack 0712bf9f not referenced in index. Can be a parallel backup job. To repair: 'rustic repair index'.
[WARN] pack 0712bf9f not referenced in index. Can be a parallel backup job. To repair: 'rustic repair index'.
[WARN] pack 0712bf9f not referenced in index. Can be a parallel backup job. To repair: 'rustic repair index'.
[WARN] pack 0712bf9f not referenced in index. Can be a parallel backup job. To repair: 'rustic repair index'.
[WARN] pack 0712bf9f not referenced in index. Can be a parallel backup job. To repair: 'rustic repair index'.
[WARN] pack 333ebde7 not referenced in index. Can be a parallel backup job. To repair: 'rustic repair index'.
[WARN] pack 47661552 not referenced in index. Can be a parallel backup job. To repair: 'rustic repair index'.
[WARN] pack aa509bfd not referenced in index. Can be a parallel backup job. To repair: 'rustic repair index'.
[WARN] pack facc363c not referenced in index. Can be a parallel backup job. To repair: 'rustic repair index'.
[00:00:00] listing packs...
[00:00:00] cleaning up packs from cache...
[00:00:43] checking packs in cache...     ████████████████████████████████████████ 163.97 MiB/163.97 MiB 3.79 MiB/s   (ETA 0s)
[00:00:00] reading snapshots...           ████████████████████████████████████████         48/48
[00:00:00] checking trees...              █░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░          2/48
[ERROR] file "home/sperber/.config/ironbar/config.toml" blob d16484e9 is missing in index
[ERROR] file "home/sperber/.config/ironbar/config.toml" blob d16484e9 is missing in index
[ERROR] file "home/sperber/.librewolf/94ufbqpk.default/addonStartup.json.lz4" blob 14256e46 is missing in index
[ERROR] file "home/sperber/.config/fish/fish_variables" blob 77db7704 is missing in index
[00:00:00] checking trees...              ████████████████████████████████████████         48/48

I then run

 ~  rustic -P slim -P MEGA repair index                                                                                                                                                                                                                                                                          1m57s  15:19
[INFO] using config /home/sperber/.config/rustic/slim.toml
[INFO] using config /home/sperber/.config/rustic/MEGA.toml
[INFO] repository rclone:MEGA:Rustic: password is correct.
[INFO] using cache at /home/sperber/.cache/rustic/378cce4ff1059d99310953312ff66e49029e0ce98472c6d4190fa10f33458952
[00:00:00] listing packs...
[00:00:00] reading index...               ████████████████████████████████████████        128/128
[00:00:00] reading pack headers           ████████████████████████████████████████          0/0

If i then run the check again, exactly the same output is shown. I rebooted my system and went through the steps again, the result is the same.

I once started the forget command and then cancelled it after a short while, so i think that is where the errors are from. However, i don't know why repair doesn't work.

[Schweber]

@aawsome might this be fixed by #1078 ?

[aawsome]

@Schweber Thanks a lot for reporting the issue! And very sorry for the late reply.

Actually, what you are reporting is a mystery for me. The check run lists all pack files, reads the index files and recognizes that there are pack files missing in the index. The repair index run lists all pack files, reads the index files and collects pack files missing in the index. Those are then read - or better the headers of those pack files. And the number of pack files headers to read is 0....

#1078 won't fix this as it does exactly the same as repair index does. It actually shares this logic with repair index.

Can you try to re-run your example with the latest nightly builds? I would like to see if you can still reproduce it there. If you still get the same behavior we have to get more debug information of what's going wrong...

[Schweber]

I compiled rev 0f3cc39 and it shows exactly the same behaviour as version 0.7.0 (rustic -V puts out rustic 0.7.0-dev now so it worked).

I ran the commands again with debug log-level, i hope this is useful to you:

rusticCheck.log
rusticRepairIndex.log

[aawsome]

@Schweber Sorry for coming back so late to this topic. I tried to reproduce the behavior using rustic v0.7.0 - but wasn't able to.

To summarize your problem:

• you have pack files which are not referenced by the index, this is reported by check
• running repair index does try to read 0 pack headers.

I tried to reproduce this by manually adding packfiles to the repository.
check reported exactly the same message you got:

`WARN] pack dcba955f not referenced in index. Can be a parallel backup job. To repair: 'rustic repair index'.

rustic repair index, however, did read this one pack file header:

[00:00:00] reading pack headers           ████████████████████████████████████████          1/1 

and afterwards, rustic check no longer complained about packs not-in-index.

So, it seems not to be a general rustic repair index problem, but either something specific to your repository setting and/or some edge case handling.

Unfortunately, the debug logs do not contain any more information here, but can you please:

• post the output of rustic --version?
• Manually check the listed pack files in your backend (under /data) to see if there is something notable about them?

Thanks a lot for your support and patience!

[Schweber]

I'm back on "mainline" so

$ rustic --version                                                                                                                           
rustic 0.7.0

Nothing stands out to me in the /data directory but i wouldn't know what to look for anyway. I append the screenshots as i didn't find a way to export them in text form:

[aawsome]

Sorry for being not precise. I actually meat the files which are bailed by check, i.e. the following files:

• /data/07/0712bf9f....
• /data/33/333ebde7....
• /data/47/47661552...
• /data/aa/aa509bfd...
• /data/fa/facc363c....

[Schweber]

Oh, i see. I post the screenshots of them:

As mentioned, i once ran the forget command and canceled it before it was complete (i don't know if this was on 13.11.2023 though). However, i wonder why rustic produced the duplicates of these files.

[aawsome]

Ok, this is interesting and could be the cause of the issue. Those seem to be completely identical files. This is usually not even possible...Which storage backend are you using here?

[aawsome]

As mentioned, i once ran the forget command and canceled it before it was complete (i don't know if this was on 13.11.2023 though). However, i wonder why rustic produced the duplicates of these files.

I can't imagine that this is caused by a forget run. forget only removes snapshot files and doesn't even touch pack files under /data.

I also wonder how these duplicates could appear. rustic would normally only try to save a file a second time if the storage backend reported an error while saving it. And as said, usually backends don't even allow to save identical duplicates.

[Schweber]

Which storage backend are you using here?

The backend is mega.nz (via rclone).

[aawsome]

So, I think this is the cause for this issue:

• check reads the file list from the backend and processes each-by-each removing them from a set of packs generated from the index. As you have duplicate packs, it bails the second (and third...) as not existing in the index, which is a false positive.
• repair index in contrast reads the file list from the backend and puts them first into a set before processing it. By this, duplicates are automatically removed. So repair index has nothing to do, which is also the correct behavior in this case.

@Schweber In general, this is something which can't be repaired by rustic, as the API to the backend assumes that there is no possibility for duplicates. You can however repair it by hand: If you can compare the files and make sure, that files with same name are identical (which seems to be), just remove the duplicates. Else, you might want to backup them just in case only one of the duplicates would work and choose the right one. Anyway, after that, check should no longer report an error.

I will add this extra check to check, so we will no longer get this irritation error message but instead something like "duplicate files in backend - please manually repair by removing the wrong or additional ones".

[aawsome]

And thanks a lot @Schweber for reporting - this is indeed a rare edge case which I haven't thought of when implementing check.

[Schweber]

In general, this is something which can't be repaired by rustic, as the API to the backend assumes that there is no possibility for duplicates.

Oh, ok. It would be nice if it could do that as rustic knows best which file is the correct/working one, maybe as a separate command like scrub. But if it's not possible then that's the way it is.

One question regarding this: what happens if i were to delete the wrong "duplicates"? Would rustic notice this during check (is it comparing hashes or just filenames?) or would this go unnoticed until an actual attempt to restore relevant files takes place?

And thanks a lot @Schweber for reporting

Sure, thank you for creating rustic!

[aawsome]

One question regarding this: what happens if i were to delete the wrong "duplicates"? Would rustic notice this during check

Yes, a check --read-data would recognize it. Besides this, you can manually compute the SHA256 of the the file content (e.g. use sh256sum). If it matches the name, the content is ok.

[Schweber]

Alright, i deleted the duplicates and the [WARN]s are gone but the [ERROR]s persist:

 ~  rustic -P slim -P MEGA repair index                                                                                                
[INFO] using config /home/sperber/.config/rustic/slim.toml
[INFO] using config /home/sperber/.config/rustic/MEGA.toml
[INFO] repository rclone:MEGA:Rustic: password is correct.
[INFO] using cache at /home/sperber/.cache/rustic/378cce4ff1059d99310953312ff66e49029e0ce98472c6d4190fa10f33458952
[00:00:00] listing packs...
[00:00:00] reading index...               ████████████████████████████████████████        254/254
[00:00:00] reading pack headers           ████████████████████████████████████████          0/0

 ~  rustic -P slim -P MEGA check                                                                                                               
[INFO] using config /home/sperber/.config/rustic/slim.toml
[INFO] using config /home/sperber/.config/rustic/MEGA.toml
[INFO] repository rclone:MEGA:Rustic: password is correct.
[INFO] using cache at /home/sperber/.cache/rustic/378cce4ff1059d99310953312ff66e49029e0ce98472c6d4190fa10f33458952
[00:00:00] checking Snapshot in cache...  ████████████████████████████████████████  74.16 KiB/74.16 KiB  19.49 MiB/s  (ETA 0s)
[00:00:00] checking Index in cache...     ████████████████████████████████████████  34.26 MiB/34.26 MiB  3.97 GiB/s   (ETA 0s)
[00:00:00] reading index...               ████████████████████████████████████████        254/254[00:00:00] listing packs...
[00:00:00] cleaning up packs from cache...
[00:01:21] checking packs in cache...     ████████████████████████████████████████ 320.31 MiB/320.31 MiB 3.92 MiB/s   (ETA 0s)
[00:00:00] reading snapshots...           ████████████████████████████████████████        124/124
[00:00:00] checking trees...              ████░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░         13/124
[ERROR] file "home/sperber/.config/ironbar/config.toml" blob d16484e9 is missing in index
[ERROR] file "home/sperber/.librewolf/94ufbqpk.default/addonStartup.json.lz4" blob 14256e46 is missing in index
[ERROR] file "home/sperber/.config/ironbar/config.toml" blob d16484e9 is missing in index
[ERROR] file "home/sperber/.config/fish/fish_variables" blob 77db7704 is missing in index
[00:00:01] checking trees...              ████████████████████████████████████████        124/124

What does that mean and how can i fix this? Or is this beyond the scope of repair index and the file is simply (why?) missing?

[aawsome]

Ok, this is a different problem, unrelated to the repair index topic. There is some data missing in your repository. The easiest way to fix it is to rerun a backup containing those files - if they haven't changed. But I suggest to just try and see if the error is gone after. If not, you cannot get the file contents, but you can repair the repository by altering the affected snapshots using repair snapshots.