Prevent Curation from re-adding an existing sorting key with a new cu…

[samuelbray32]

Description

Calling the spikesorting_pipeline_populator() again with the same parameters after already running resulted in many duplicate entries. This was because insert_curation() called with an existing key will create a new entry with a iterated curation_id. This new entry then runs in all susequent steps of the pipeline, using up compute time and memory.

This PR avoids this by checking for existing entries in Curation for the sort key before calling insert_curation()

Checklist:

• This PR should be accompanied by a release: (yes/no/unsure)
• (If release) I have updated the CITATION.cff
• I have updated the CHANGELOG.md
• I have added/edited docs/notebooks to reflect the changes

[CBroz1]

It looks like this check is already present in the insert curation func, which we're calling in the populator with the default parent curation id. Does this new check catch cases the sub-func doesn't?

[samuelbray32]

It looks like this check is already present in the insert curation func, which we're calling in the populator with the default parent curation id. Does this new check catch cases the sub-func doesn't?

Thanks for the catch. I'll start digging again

[samuelbray32]

Next idea:

Populating AutomaticCuration creates a new curation_id. If this function is run again, that curation_id will be incorporated in
curation_keys = [ {**k, "waveform_params_name": waveform_params_name} for k in (Curation() & sort_dict).fetch("KEY") ]

And then get run through the remaining pipeline steps.

Propose restricting this key list to curation_id=0 to avoid this. (My todo).

[edeno]

Should this be closed or is this still being worked on?

[samuelbray32]

Should this be closed or is this still being worked on?

The last commit fixes the initial issue. This isn't updated for the v1 spikesorting pipeline though. If that doesn't need to happen in the current PR this is good to go