Set Scylla user task TTL when using native Scylla API #4238
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
We need to extend run progress with Scylla task ID. Moreover, we need it to be a part of the clustering key Otherwise, we won't be able to store multiple run progresses with the same remote sstable dir and host in the DB. That's why we need to re-create the table. It also allows us to rename manifest path and versioned progress to more suitable names. The downside is that users will lose their restore progress history on upgrade, but this shouldn't harm anybody if communicated properly.
It's good to have it specified in our codebase. E.g. the knowledge about the TOC component will be necessary for native Scylla restore.
This commit extends SSTable structure with its TOC component, which is needed when using Scylla restore API. Moreover, it introduces batch types, which are also needed for deciding, whether given batch can be restored with Scylla restore API or the Rclone API. It also makes sure that all SSTables within the same batch belong to the same batch type.
It's better to do it at the start so that: - we don't need to do it per restore type - we don't create and immediately return the batch
With native Scylla restore, we can gradually update restored bytes progress.
This commit adds new metrics: - restored_bytes - restore_duration which are counterparts of the similar downloaded and streamed metrics. It also adds new restore state (RestoreStateNativeRestore) which describes the usage of native Scylla restore
This commit makes TestRestoreTablesPreparationIntegration check for the proper API used for restore.
This commit makes TestRestoreTablesPreparationIntegration check for proper Scylla user task TTL set before/during/after restore.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Rebased on #4192.
Fixes #4158