Problem: Replicated AIPs cannot be packaged in a different format from the original

[ablwr]

Please describe the problem you'd like to be solved
Users should be able to create replicated AIPs that are packaged in a format different from the original AIP.

Users of Archivematica that are storing their AIPs uncompressed and unwrapped may want one or more replicated copies of their AIPs to move to a storage location that is better supported through a single, wrapped copy of the AIP (such as tape-based storage systems, or possibly S3-based storage).

And likewise, the reverse may be true, where it may favor a potential workflow if Archivematica's default wrapped and compressed AIPs could have a replicated copy that is unwrapped and uncompressed.

In order to achieve this, Archivematica should have an additional feature that makes it able to store one replica as an unpacked AIP and at least one replica stored as a packaged TAR file (uncompressed).

Describe the solution you'd like to see implemented
Some work has been done on this feature, predating my time on this project: artefactual/archivematica-storage-service#574

Below is a Gherkin-style template that can be used as acceptance criteria:

Feature: Store replicated copy of AIP using different packaging algorithm
  Scenario: The Archivematica Storage Service is configured to replicate AIPs to multiple Locations
     Given: The primary storage location has a replicator location connected to it
    Given: The replicated storage location is configured to wrap uncompressed packages as a single TAR file
 
 When: An Archivematica stores an AIP
  Then: One AIP will get stored in the Storage Service’s predefined location
  And: This AIP will get replicated using the Replication service in AMSS
    Then: One replicated AIP will get stored in another disk-based location
      Then: The replicated AIP will get packaged as a TAR
      And: The AIP packaged as a TAR will be transferred to a Tape Archive

Describe alternatives you've considered

• Not having Archivematica manage this functionality, but rather rely on something like a cron job to achieve this feature.
• Retroactively wrap AIPs so that a (uncompressed but wrapped) workflow is used for the original and all other replicated AIPs
• Having Enduro manage this feature through a post-processing hook.

Additional context
This is a sponsored request.

---

For Artefactual use:

Before you close this issue, you must check off the following:

• All pull requests related to this issue are properly linked
• All pull requests related to this issue have been merged
• A testing plan for this issue has been implemented and passed (testing plan information should be included in the issue body or comments)
• Documentation regarding this issue has been written and merged
• Details about this issue have been added to the release notes

[ablwr]

@tw4l For this feature, since users cannot download AIPs from the write-only replica, can we hide the "Download" option in the Storage Service GUI? Otherwise, seems to be working fine across replicas. When I do click "Download" for this replica, I get an error, which is expected. But it'd be better to not have the ability to click a link at all.

Should I make a new Issue or should there be a new PR assigned to this one?

[tw4l]

Thanks @ablwr! I'll look into hiding the "Download" option in the GUI for this option. Since this is a refinement of the same issue, we can continue to use this issue and link any subsequent PRs here.

[tw4l]

After feedback from @sromkey we opted not to hide the download links - see discussion starting here: artefactual/archivematica-storage-service#595 (comment).

[tw4l]

artefactual/archivematica-storage-service#597 introduces tweaks to the Storage Service which enable creating replicas in a new replicator location, so that users who want to replicate their AIPs into a Replicator location setup in the new OfflineReplicaStaging space can do so without needing to touch their other AIP replicas.

[replaceafill]

I've confirmed that:

• compressed AIPs are replicated as 7z files and uncompressed AIPs are replicated as tar files in the Write-Only Replica Staging on Local Filesystem space
• there are no leftovers in the processing directory
• trying to download a replica from this space produces the error Write-Only Offline Staging does not implement fetching packages
• the layout of replicator's directory contains no quad directories, only the replica files

replaceafill@eluk:~/dev/archivematica/archivematica/hack$ sudo tree ~/.am/am-pipeline-data/www/
/home/replaceafill/.am/am-pipeline-data/www/
├── AIPsStore
│   ├── 2979
│   │   └── 0589
│   │       └── eb0e
│   │           └── 4a82
│   │               └── ba25
│   │                   └── fac6
│   │                       └── 1519
│   │                           └── feba
│   │                               └── demo-29790589-eb0e-4a82-ba25-fac61519feba.7z
│   ├── 33b7
│   │   └── 3859
│   │       └── 659e
│   │           └── 46b8
│   │               └── 8527
│   │                   └── 508c
│   │                       └── 363b
│   │                           └── 6ae5
│   │                               └── demo-uncompressed-33b73859-659e-46b8-8527-508c363b6ae5
│   │                                   ├── bag-info.txt
│   │                                   ├── bagit.txt
│   │                                   ├── data
│   │                                   │   ├── logs
│   │                                   │   │   ├── fileFormatIdentification.log
│   │                                   │   │   ├── filenameChanges.log
│   │                                   │   │   ├── FileUUIDs.log
│   │                                   │   │   └── transfers
│   │                                   │   │       └── demo-uncompressed-e631788c-4043-4193-9236-015c2d536880
│   │                                   │   │           └── logs
│   │                                   │   │               ├── fileFormatIdentification.log
│   │                                   │   │               ├── filenameChanges.log
│   │                                   │   │               └── FileUUIDs.log
│   │                                   │   ├── METS.33b73859-659e-46b8-8527-508c363b6ae5.xml
│   │                                   │   ├── objects
│   │                                   │   │   ├── artwork
│   │                                   │   │   │   ├── MARBLES-71a80f92-cbed-40aa-97a2-0a8f7dbcac08.tif
│   │                                   │   │   │   ├── MARBLES.TGA
│   │                                   │   │   │   └── Montreal.png
│   │                                   │   │   ├── beihai-d1ce9e14-c23b-45e8-b05a-20091bd3752c.tif
│   │                                   │   │   ├── beihai.tif
│   │                                   │   │   ├── bird-6bd02c26-738d-4631-b04f-be651e0c29b4.wav
│   │                                   │   │   ├── bird.mp3
│   │                                   │   │   ├── metadata
│   │                                   │   │   │   └── transfers
│   │                                   │   │   │       └── demo-uncompressed-e631788c-4043-4193-9236-015c2d536880
│   │                                   │   │   │           ├── checksum.md5
│   │                                   │   │   │           ├── checksum.sha1
│   │                                   │   │   │           ├── checksum.sha256
│   │                                   │   │   │           ├── checksum.sha512
│   │                                   │   │   │           ├── identifiers.json
│   │                                   │   │   │           ├── metadata.csv
│   │                                   │   │   │           └── rights.csv
│   │                                   │   │   ├── ocr-image.png
│   │                                   │   │   ├── piiTestDataCreditCardNumbers.txt
│   │                                   │   │   ├── submissionDocumentation
│   │                                   │   │   │   └── transfer-demo-uncompressed-e631788c-4043-4193-9236-015c2d536880
│   │                                   │   │   │       ├── Agreement-Gift-MBRS-project.pdf
│   │                                   │   │   │       └── METS.xml
│   │                                   │   │   └── View_from_lookout_over_Queenstown_towards_the_Remarkables_in_spring.jpg
│   │                                   │   └── README.html
│   │                                   ├── manifest-sha256.txt
│   │                                   └── tagmanifest-sha256.txt
│   ├── c37f
│   │   └── eaa2
│   │       └── ed35
│   │           └── 488f
│   │               └── b98b
│   │                   └── 1502
│   │                       └── aab8
│   │                           └── d612
│   │                               └── demo-offline-c37feaa2-ed35-488f-b98b-1502aab8d612.7z
│   └── transferBacklog
│       ├── arrange
│       └── originals
├── DIPsStore
└── offline-replicas
    ├── demo-offline-9d85525f-9919-402e-8bd9-7c1f566965c9.7z
    └── demo-uncompressed-5da3af62-e799-432a-b82f-48b30ec04552.tar