Enhancement: Add package name as configurable value to post-store AIP, AIC and DIP callbacks

[tw4l]

Please describe the problem you'd like to be solved

At Concordia University Library and in collaboration with EPrints Services and the community of EPrints users, we are currently working on an EPrints-Archivematica integration via an EPrints-Archivematica export plugin. This plugin will export newly added or modified eprints in an EPrints instance, along with their metadata and object checksums, in a package format that is optimized for Archivematica. We are planning to use Automation Tools to then pick up these packages and ingest them through an automated Archivematica pipeline.

We would like to send a notification back to EPrints following successful storage of the AIP which includes the resulting AIP UUID using the existing post-store AIP callback feature. In order for EPrints to write this UUID into the correct place in the EPrints database, the callback needs to include the EPrintsArchivematicaDatasetID (the primary key in the table in EPrints which keeps track of the exports created by the plugin).

Describe the solution you'd like to see implemented

After consultation with Sara and Ross, we think that the simplest and most generic solution is to include a new configurable <package_name> value to the URI/body of the post-store AIP/AIC/DIP callback. For the EPrints integration, we will then include the EPrintsArchivematicaDatasetID in the transfer name so that the resulting post-store callback contains both pieces of needed information.

This solution should work with other originating systems as well, as any system could use its originating ID as a transfer name and then receive both that originating ID and the resulting AIP UUID in the post-store callback.

Describe alternatives you've considered

• Including the EPrintsArchivematicaDatasetID with the transfer in a file like metadata.csv or identifiers.json and then adding code to the Storage Service to be able to look up this value and include it in the post-store callback.
• Using the Storage Service callback as-is and looking up the package name using the Archivematica API Ingest status endpoint to look up the package name. We'd prefer to avoid this option as it complicates integration from the perspective of the originating system (in this case, EPrints).

Additional context

Relevant Issues:

• #147
• Relevant Issue in the EPrintsArchivematica repository

Edit: Added link to relevant issue in EPrintsArchivematica repository

---

For Artefactual use:

Before you close this issue, you must check off the following:

• All pull requests related to this issue are properly linked
• All pull requests related to this issue have been merged
• A testing plan for this issue has been implemented and passed (testing plan information should be included in the issue body or comments)
• Documentation regarding this issue has been written and merged
• Details about this issue have been added to the release notes

[ross-spencer]

Hi @timothyryanwalsh This is really starting to take shape. Have a look at the following for extending the Storage Service, and we're here to CR!!

• Modifying package.py.
• And then adding some tests following the examples here: existing example.

We've a few more items to triage for 1.11 just pinging @sromkey here and cc. @sallain for our next backlog grooming meeting as something we should be able to add to this release pretty easily.

[tw4l]

Hi @ross-spencer - PR is submitted and ready for code review!

[ross-spencer]

I was looking up areas where documentation may or may not need to be updated with this enhancement. A by-product of which was following up with artefactual/archivematica-storage-service#502 to update all of our pip requirements, which may or may not be linked to this issue.

Another by-product is I have documented my previous testing. I have also tested this feature, and from the notes in the attached gist I feel it's looking good. Should we test again closer to 1.11 then the notes might be beneficial to whomever might do that. It's one approach, I'd be very welcome to hear other approaches that allows us to do more generally in future.

As for documentation, then this is related to: #947 where we might elect to expand on what is there with a worked example, e.g.

given a configuration where perhaps you'd like JSON to consume:

{
   "PackageID": "<package_uuid>",
   "PackageName": "<package_name>"
}

You'd get JSON that looks like the following to consume:

{
   "PackageID": "fd6a3f60-1431-4ca2-b146-ca2a57f438c9",
   "PackageName": "20123_New_transfer_002.001.124"
}

Additional notes that may be useful to documentation are in the gist.

[tw4l]

I have confirmed that the feature is operating as expected in branch qa/1.x (Archivematica) and qa/0.x (Storage Service) for the AM 1.11/SS 0.16 release. I used a similar process to what @ross-spencer has linked to in the gist above in my local Docker dev environment:

• Start server on host's localhost (in my case, using node) that will accept/log arbitrary incoming requests
• Set up post-store AIP callback in Storage Service, using <package_name> and <package_id> placeholders in both the body and URI
• Start transfer with name callback-test.0.fakeext on an automated pipeline
• Confirm that post-store callback is successfully sent and all placeholders are replaced with expected values after the AIP is successfully stored
• For good measure, repeat the above but using the "Post-store DIP" and "Post-store AIC" event types

The callbacks executed successfully with no observed side effects for all three event types.

The Storage Service documentation has been updated with additional information about how to configure and use the post-store callbacks. See: artefactual/archivematica-storage-service-docs#35

FYI for @ross-spencer and others who may find this helpful: When running the Storage Service in Docker on Linux, setting the base URI for the callback to http://172.17.0.1:<port> enables the Storage Service to send its callback to the host's localhost on port <port>. I believe http://host.docker.internal will do the same with Docker for Mac.