Aerospike sink: resumability #2399
Closed
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Jesse-Bakker
changed the title
Some sources that don't have transactions might use a last-update-time as a txid, and not have commits for every different txid as a result. Therefore, we're a bit more relaxed about it. Because we filter on OperationId when writing ops, this should be fine.
(Hey, that rhymes)
|
Superseeded by #2437 |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This implements resuming for the aerospike sink in the following way:
When restarting, the commit OperationId is returned as the last operation id.
This might be behind (because aerospike semantics and because we write it asynchronously),
but we catch that by only applying operations if the record they apply to has
an OperationId that is smaller than the applied operation. This means the
commit OperationId is only there to get a reasonably tight lower bound on the
OperationId, as it's not needed for correctness.