De-duplicate checksum verification

[jrwdunham]

Fixes #918

In short, removes the once-per-file verifyPREMISChecksums_v0.0 micro-service and replaces it with once-per-SIP equivalent functionality in a new verifyAIP_v1.0 client script (which supersedes verifyAIP_v0.0). Question: I have deleted verifyAIP_v0.0 because other superseded client scripts appear to have been deleted. Should it be retained?

The new verifyAIP_v1.0 parses the BagIt manifest file for the checksums that bag has just calculated instead of needlessly re-calculating them.

The functional consequences of this change are as follows.

1. AIP METS files will no longer have fixity check PREMIS events for each file.

2. The AIP pointer file will now contain a single fixity check PREMIS event for the entire AIP:

   <mets:mdWrap MDTYPE="PREMIS:EVENT">
     <mets:xmlData>
       <premis:event xmlns:premis="info:lc/xmlns/premis-v2" xsi:schemaLocation="info:lc/xmlns/premis-v2 http://www.loc.gov/standards/premis/v2/premis-v2-2.xsd" version="2.2">
         <premis:eventIdentifier>
           <premis:eventIdentifierType>UUID</premis:eventIdentifierType>
           <premis:eventIdentifierValue>bdf5c772-1923-4241-a88b-1f949ab542c2</premis:eventIdentifierValue>
         </premis:eventIdentifier>
         <premis:eventType>fixity check</premis:eventType>
         <premis:eventDateTime>2018-03-23T16:57:37.602859+00:00</premis:eventDateTime>
         <premis:eventDetail>program="python, bag"; module="hashlib.sha256()"</premis:eventDetail>
         <premis:eventOutcomeInformation>
           <premis:eventOutcome>Pass</premis:eventOutcome>
           <premis:eventOutcomeDetail>
             <premis:eventOutcomeDetailNote>
               All 3 checksums generated at start of transfer match those generated by BagIt (bag).
             </premis:eventOutcomeDetailNote>
           </premis:eventOutcomeDetail>
         </premis:eventOutcomeInformation>
         <premis:linkingAgentIdentifier>
           <premis:linkingAgentIdentifierType>preservation system</premis:linkingAgentIdentifierType>
           <premis:linkingAgentIdentifierValue>Archivematica-1.7</premis:linkingAgentIdentifierValue>
         </premis:linkingAgentIdentifier>
         <premis:linkingAgentIdentifier>
           <premis:linkingAgentIdentifierType>repository code</premis:linkingAgentIdentifierType>
           <premis:linkingAgentIdentifierValue>test</premis:linkingAgentIdentifierValue>
         </premis:linkingAgentIdentifier>
       </premis:event>
     </mets:xmlData>
   </mets:mdWrap>
   

3. AIPs without pointer files will have no record of the SIP/AIP-level fixity check. This was determined to be an acceptable trade-off and should be remedied once we [begin creating pointer files for uncompressed AIPs] (Problem: uncompressed AIPs need pointer files archivematica-storage-service#324), which is motivated by other considerations as well.

[sevein]

https://www.python.org/dev/peps/pep-0008/#package-and-module-names

Modules should have short, all-lowercase names. Underscores can be used in the module name if it improves readability. Python packages should also have short, all-lowercase names, although the use of underscores is discouraged.

[sevein]

Can you move this call inside the if __name__ == '__main__': block? I find it's more explicit and clearer. If there was a case where someone is using this module then the caller should be responsible to initialize Django. Probably not very important, but I've been doing it this way recently.

[jrwdunham]

ok

[sevein]

The default mode is ok? Just checking.

[jrwdunham]

Seems to be ok. Extraction works

[sevein]

Nice way to put it with the examples, that's useful :)

[jrwdunham]

I didn't do it, but I'll take the compliment

[sevein]

One could argue that it's not verify_checksums's concern to know about exit codes - it'd be more pythonic to have it raise an exception right? Feel free to ignore this. Just a random thought, I've done the same so many times.

[jrwdunham]

Yeah, I think you're right. I'll change.

[sevein]

Do we want to make the script report an error if the clean up did not succeed? You may want to trap OSError if that's not the case.

[jrwdunham]

sounds good

[sevein]

Sorry I didn't realize that it wasn't really your code what I was looking at!