DM-35396: Add provenance hooks for datastore to use #1147
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
2 tasks
Codecov ReportAttention: Patch coverage is
✅ All tests successful. No failed tests found.
Additional details and impacted files@@ Coverage Diff @@
## main #1147 +/- ##
=======================================
Coverage 89.47% 89.47%
=======================================
Files 366 367 +1
Lines 49115 49185 +70
Branches 5955 5960 +5
=======================================
+ Hits 43944 44007 +63
- Misses 3753 3759 +6
- Partials 1418 1419 +1 ☔ View full report in Codecov by Sentry. |
timj
force-pushed
the
tickets/DM-35396
branch
from
a8247c8 to
880b214
Compare
2 tasks
timj
force-pushed
the
tickets/DM-35396
branch
from
880b214 to
ff6ea4a
Compare
TallJimbo
approved these changes
Jan 27, 2025
timj
force-pushed
the
tickets/DM-35396
branch
from
0a3bb85 to
7ddf988
Compare
Open
2 tasks
timj
force-pushed
the
tickets/DM-35396
branch
from
7ddf988 to
bcdbff4
Compare
andy-slac
approved these changes
Jan 31, 2025
| @@ -453,11 +454,11 @@ def dimensions(self) -> DimensionUniverse: | |||
| # Docstring inherited. | |||
| return self._dimensions | |||
|
|
|||
| def put(self, obj: Any, ref: DatasetRef, /) -> DatasetRef: | |||
| def put(self, obj: Any, ref: DatasetRef, /, provenance: DatasetProvenance | None = None) -> DatasetRef: | |||
There was a problem hiding this comment.
Suggested change
| def put(self, obj: Any, ref: DatasetRef, /, provenance: DatasetProvenance | None = None) -> DatasetRef: | |
| def put(self, obj: Any, ref: DatasetRef, /, *, provenance: DatasetProvenance | None = None) -> DatasetRef: |
to make it consistent with LimitedButler?
| def put(self, inMemoryDataset: Any, datasetRef: DatasetRef) -> None: | ||
| def put( | ||
| self, inMemoryDataset: Any, datasetRef: DatasetRef, provenance: DatasetProvenance | None = None | ||
| ) -> None: | ||
| raise NotImplementedError("This is a no-op datastore that can not access a real datastore") | ||
|
|
||
| def put_new(self, in_memory_dataset: Any, ref: DatasetRef) -> Mapping[str, DatasetRef]: |
There was a problem hiding this comment.
Hmm, we should probably drop put_new if we are not using it. It's not tested and is probably broken already. Not on this ticket, of course.
| @@ -169,6 +174,7 @@ def __init__( | |||
| ref: DatasetRef, | |||
| write_parameters: Mapping[str, Any] | None = None, | |||
| write_recipes: Mapping[str, Any] | None = None, | |||
| provenance: DatasetProvenance | None = None, | |||
There was a problem hiding this comment.
Does it need to be a constructor argument, could you pass it to write instead?
Comment on lines
+84
to
+86
| if dataset_id not in self.extras: | ||
| self.extras[dataset_id] = {} | ||
| self.extras[dataset_id].update(extra) |
There was a problem hiding this comment.
Maybe
self.extras.setdefault(dataset_id, {}).update(extra)
?
timj
force-pushed
the
tickets/DM-35396
branch
from
bcdbff4 to
37dd1a7
Compare
No-op in the base class but can be used by subclasses to add provenance to composites that are to be disassembled.
timj
force-pushed
the
tickets/DM-35396
branch
from
f6b4efc to
afb16c5
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Checklist
doc/changesconfigs/old_dimensions