Restore from cold storage errors with: The operation is not valid for the object's storage class

[philipmw]

I set up rustic with a pair of S3 buckets, one hot one cold. I created a backup, and now I am trying to restore it.

My rustic.toml:

[[backup.snapshots]]
sources = ["/"]

[repository]
repository = "opendal:s3"
repo-hot = "opendal:s3"

[repository.options]
append_only = "true"
allow_anonymous = "false"
region = "us-west-2"

[repository.options-cold]
bucket = "pmw-backups-laptop-rustic-cold"
default_storage_class = "DEEP_ARCHIVE"

[repository.options-hot]
bucket = "pmw-backups-laptop-rustic-hot"

Here's my attempt to restore the data, using rustic 0.9.5:

% ./rustic restore latest ~/rustic-latest-snapshot -p repo-password.dat
[INFO] using config ./rustic.toml
[INFO] repository opendal:pmw-backups-laptop-rustic-cold#opendal:pmw-backups-laptop-rustic-hot: password is correct.
[INFO] using cache at /Users/pmw/Library/Caches/rustic/7e2aeced83b5ea2923e9cd1f9391d762b015abedd588d6394bb24568c228fdea
[00:00:00] reading index...               ████████████████████████████████████████          1/1                                                                                                                                                      [00:00:00] getting latest snapshot...     ████████████████████████████████████████          1/1                                                                                                                                                      [00:00:00] collecting file information...                                                                                                                                                                                                            [INFO] total restore size: 1.4 GiB
Files:  38904 to restore, 564 unchanged, 0 verified, 0 to modify, 0 additional
Dirs:   0 to restore, 6761 to modify, 0 additional
[00:00:00] restoring file contents...     ░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░        0 B/1.36 GiB   0 B/s        (ETA -)                                                                                                                        
[WARN] service=s3 name=pmw-backups-laptop-rustic-cold path=data/3e/3ec9d189e90d64c9da21f1bd9e5e86c006ee2a409267a1f23e395ed71240b816: read failed PermissionDenied (persistent) at read, context: { uri: https://s3.us-west-2.amazonaws.com/pmw-backups-laptop-rustic-cold/data/3e/3ec9d189e90d64c9da21f1bd9e5e86c006ee2a409267a1f23e395ed71240b816, response: Parts { status: 403, version: HTTP/1.1, headers: {"x-amz-request-id": "XGRZTREGWXR7D3XE", "x-amz-id-2": "RjEefVOLLLuTTHAkk0OfqredLCx44SgElyKMq5iO45VnUIAXY8HADygm5GVMyatrBUA8zBn5i+A=", "content-type": "application/xml", "transfer-encoding": "chunked", "date": "Mon, 09 Dec 2024 20:08:13 GMT", "server": "AmazonS3"} }, service: s3, path: data/3e/3ec9d189e90d64c9da21f1bd9e5e86c006ee2a409267a1f23e395ed71240b816, range: 1671612-1671839 } => S3Error { code: "InvalidObjectState", message: "The operation is not valid for the object's storage class", resource: "", request_id: "XGRZTREGWXR7D3XE" }
[WARN] service=s3 name=pmw-backups-laptop-rustic-cold path=data/04/0436190337a3424624733ab63715384d80e71401362e54589738fc7e8dbcf5fc: read failed PermissionDenied (persistent) at read, context: { uri: https://s3.us-west-2.amazonaws.com/pmw-backups-laptop-rustic-cold/data/04/0436190337a3424624733ab63715384d80e71401362e54589738fc7e8dbcf5fc, response: Parts { status: 403, version: HTTP/1.1, headers: {"x-amz-request-id": "XGRW997R9H8MDZV2", "x-amz-id-2": "PGXL3gBYx80NWQDnCjUgcDr3vkQ6dBBAetooiKJvD0bxieqO/56Bz8NNoV6j2tT/hpjZwlVt+z0=", "content-type": "application/xml", "transfer-encoding": "chunked", "date": "Mon, 09 Dec 2024 20:08:13 GMT", "server": "AmazonS3"} }, service: s3, path: data/04/0436190337a3424624733ab63715384d80e71401362e54589738fc7e8dbcf5fc, range: 0-37157614 } => S3Error { code: "InvalidObjectState", message: "The operation is not valid for the object's storage class", resource: "", request_id: "XGRW997R9H8MDZV2" }
Well, this is embarrassing.

rustic-rs had a problem and crashed. To help us diagnose the problem you can send us a crash report.

We have generated a report file at "/var/folders/45/kzx7hb411nb90kv5_f_phw6c0000gq/T/report-963d21b1-125f-48e2-a7c0-463adc141ff2.toml". Submit an issue or email with the subject of "rustic-rs Crash Report" and include the report as an attachment.

The generated report:

name = "rustic-rs"
operating_system = "Mac OS 15.1.1 [64-bit]"
crate_version = "0.9.5"
explanation = """
Panic occurred in file '/Users/runner/.cargo/registry/src/index.crates.io-6f17d22bba15001f/rustic_core-0.7.2/src/commands/restore.rs' at line 532
"""
cause = '''
called `Result::unwrap()` on an `Err` value: RusticError { kind: Backend, guidance: "Partially reading file `{path}` failed in the backend. Please check if the given path is correct.", docs_url: None, error_code: None, ask_report: false, existing_issue_urls: [], new_issue_url: None, context: [("path", "data/3e/3ec9d189e90d64c9da21f1bd9e5e86c006ee2a409267a1f23e395ed71240b816"), ("type", "Pack"), ("id", "3ec9d189"), ("offset", "1671612"), ("length", "228")], source: Some(PermissionDenied (persistent) at read => S3Error { code: "InvalidObjectState", message: "The operation is not valid for the object's storage class", resource: "", request_id: "XGRZTREGWXR7D3XE" }

Context:
   uri: https://s3.us-west-2.amazonaws.com/pmw-backups-laptop-rustic-cold/data/3e/3ec9d189e90d64c9da21f1bd9e5e86c006ee2a409267a1f23e395ed71240b816
   response: Parts { status: 403, version: HTTP/1.1, headers: {"x-amz-request-id": "XGRZTREGWXR7D3XE", "x-amz-id-2": "RjEefVOLLLuTTHAkk0OfqredLCx44SgElyKMq5iO45VnUIAXY8HADygm5GVMyatrBUA8zBn5i+A=", "content-type": "application/xml", "transfer-encoding": "chunked", "date": "Mon, 09 Dec 2024 20:08:13 GMT", "server": "AmazonS3"} }
   service: s3
   path: data/3e/3ec9d189e90d64c9da21f1bd9e5e86c006ee2a409267a1f23e395ed71240b816
   range: 1671612-1671839
), severity: None, status: None, backtrace: Some(<disabled>) }'''
method = "Panic"
backtrace = """
   0:        0x102b56180 - __mh_execute_header
   1:        0x102b55ec8 - __mh_execute_header
   2:        0x102c73954 - __mh_execute_header
   3:        0x1036d35ec - __mh_execute_header
   4:        0x1037c8abc - __mh_execute_header
   5:        0x1037c87a4 - __mh_execute_header
   6:        0x1037c8714 - __mh_execute_header
   7:        0x1037c8708 - __mh_execute_header
   8:        0x103900bdc - __mh_execute_header
   9:        0x103900ec4 - __mh_execute_header
  10:        0x103624704 - __mh_execute_header
  11:        0x103915c04 - __mh_execute_header
  12:        0x102db0800 - __mh_execute_header
  13:        0x102db0468 - __mh_execute_header
  14:        0x1037cca1c - __mh_execute_header
  15:        0x191ef72e4 - __pthread_deallocate
"""

I also tried passing the --warm-up argument, but rustic dies with the same error.

The S3 files data/04/0436190337a3424624733ab63715384d80e71401362e54589738fc7e8dbcf5fc and data/3e/3ec9d189e90d64c9da21f1bd9e5e86c006ee2a409267a1f23e395ed71240b816 do exist in the cold bucket. Their storage class is "Glacier Deep Archive".

[aawsome]

Hi @philipmw, thanks for your question - I moved this to Discussions as I think it is better suited here...

Seem you set up everything correctly to store backups in your hot/cold storage. However, to access the data, it needs to be warmed up - and how to warm up has to be configured, too. Without a warm-up-configuration rustic (silently) assumes that no warm-up is needed to access the cold data.

For an example how to set up warm-up-command and warm-up-wait for AWS glacier, see the last article about Glacier in https://rustic.cli.rs/docs/stories.html. Actually we should add such an example to the example config profiles in https://github.com/rustic-rs/rustic/tree/main/config

[aawsome]

Side note: There exists warm-up and warm-up-command - both can be used to warm-up the data in the cold storage. The first one however only tries to access the data (because there are some storage providers where such an action suffices to start the warm-up). The second calls the given command in order to warm-up and this is what we need for AWS glacier.
In future rustic versions, we might be able to use the S3 API automatically to warm-up by converting to another storage class. But we use openDAL to do the access work and there this is not (yet) supported.