feat(backend): reprocessing mechanism and processing pipeline versions

[chaoran-chen]

resolves #846
preview URL: https://reprocessing.loculus.org

Summary

• Processing pipelines must now specify their version when they retrieve or submit sequences.
• The data from a newer processing pipeline will be used when all sequences that had been successfully processed by the current pipeline were also successfully processed by the newer pipeline.

PR Checklist

• All necessary documentation has been adapted.
• The implemented feature is covered by an appropriate test.
• Adapt processing pipelines
  • Dummy pipeline
  • Nextclade pipeline (thanks to Cornelius!)

[fengelniederhammer]

Please mind #1500. Maybe rebase this on top of that to prevent further merge conflicts? The benefit is simpler code.

[corneliusroemer]

Can you describe what happens when the pipeline version increases:

• which data do you send preferentially, new entries that haven't been processed at all, or those that need reprocessing? It would be good to send new sequences preferentially, to not cause delays for current submitters.
• you say reprocessing results are only accepted if all reprocessed data doesn't error, so what happens with new data when there is an error in reprocessed sequences? Will we hence need to have two versions running in parallel to be able to have new submissions be processed on case the new version errors?

[corneliusroemer]

Pipeline version is the entirety of the processing pipeline, not just the processing image but also its configuration in values.yaml

In order to have multiple pipeline versions running for upgrading/testing, we need to adjust the k8s configuration to allow deployment of multiple pipelines, not just one as is the case at the moment.

Is the pipeline version surfaced in Silo Export? I suppose it should be to make it verifiable by end users.

[corneliusroemer]

Any clues about how to test this stuff? I suppose any tests could live entirely in the backend (unit) tests with mocked pipeline requests.

On the other hand, it's not so bad to test manually: one just makes requests with a pipeline with version X, then more requests with version X+1 and sees what the backend does. Extensive debug/trace logs would be super useful there.

[chaoran-chen]

(Cornelius and I had a short call and clarified the questions.)

[chaoran-chen]

@corneliusroemer, you're right, the error handling was wrong! Now, it should throw a 422.

[theosanderson]

(looks really nice!)

[theosanderson]

Cool! Sorry, I didn't approve before because I thought there was still un-resolved stuff from Cornelius, but I was misunderstanding.