feat(ydbcp): support ttl for backups

[ulya-sidorina]

No description provided.

[github-actions]

📝 Test results

ydbcp

33 tests (33 passed)

%%{init: {"theme":"base","themeVariables":{"fontFamily":"monospace","pieSectionTextSize":"24px","darkMode":true,"pie1":"#2da44e"}}}%%
pie showData
"Passed" : 33

Loading

📦 Package	🟢 Passed	🔴 Failed	🟡 Skipped	⏳ Duration
🟡 ydbcp/cmd/integration	0	0	0	0ms
🧪 Tests (none) 🖨️ Output (none)
🟡 ydbcp/cmd/ydbcp	0	0	0	0ms
🧪 Tests (none) 🖨️ Output (none)
🟡 ydbcp/internal/auth	0	0	0	0ms
🧪 Tests (none) 🖨️ Output (none)
🟢 ydbcp/internal/backup_operations	2	0	0	1034ms
🧪 Tests 🟢TestEndpointSecureValidation 🟢TestEndpointValidation 🖨️ Output === RUN TestEndpointValidation --- PASS: TestEndpointValidation (0.00s) === RUN TestEndpointSecureValidation --- PASS: TestEndpointSecureValidation (0.00s)
🟡 ydbcp/internal/config	0	0	0	0ms
🧪 Tests (none) 🖨️ Output (none)
🟡 ydbcp/internal/connectors/client	0	0	0	0ms
🧪 Tests (none) 🖨️ Output (none)
🟡 ydbcp/internal/connectors/db	0	0	0	0ms
🧪 Tests (none) 🖨️ Output (none)
🟢 ydbcp/internal/connectors/db/yql/queries	5	0	0	0ms
🧪 Tests 🟢TestQueryBuilder_CreateBackupSchedule 🟢TestQueryBuilder_CreateCreate 🟢TestQueryBuilder_Read 🟢TestQueryBuilder_UpdateCreate 🟢TestQueryBuilder_UpdateUpdate 🖨️ Output === RUN TestQueryBuilder_Read 2024-09-23T20:45:10.628Z DEBUG queries/read.go:144 read query {"yql": "SELECT column1, column2, column3 FROM table1 WHERE (column1 = $param0 OR column1 = $param1) AND (column2 = $param2 OR column2 = $param3)"} --- PASS: TestQueryBuilder_Read (0.00s) === RUN TestQueryBuilder_UpdateUpdate 2024-09-23T20:45:10.629Z DEBUG queries/write.go:471 write query {"yql": "UPDATE Backups SET status = $status_0, message = $message_0 WHERE id = $id_0;\nUPDATE Operations SET status = $status_1, message = $message_1 WHERE id = $id_1"} --- PASS: TestQueryBuilder_UpdateUpdate (0.00s) === RUN TestQueryBuilder_CreateCreate 2024-09-23T20:45:10.630Z DEBUG queries/write.go:471 write query {"yql": "UPSERT INTO Backups (id, container_id, database, endpoint, s3_endpoint, s3_region, s3_bucket, s3_path_prefix, status, message, size, initiated, created_at) VALUES ($id_0, $container_id_0, $database_0, $endpoint_0, $s3_endpoint_0, $s3_region_0, $s3_bucket_0, $s3_path_prefix_0, $status_0, $message_0, $size_0, $initiated_0, $created_at_0);\nUPSERT INTO Operations (id, type, status, message, initiated, created_at, container_id, database, endpoint, backup_id, operation_id) VALUES ($id_1, $type_1, $status_1, $message_1, $initiated_1, $created_at_1, $container_id_1, $database_1, $endpoint_1, $backup_id_1, $operation_id_1)"} --- PASS: TestQueryBuilder_CreateCreate (0.00s) === RUN TestQueryBuilder_UpdateCreate 2024-09-23T20:45:10.631Z DEBUG queries/write.go:471 write query {"yql": "UPDATE Backups SET status = $status_0, message = $message_0 WHERE id = $id_0;\nUPSERT INTO Operations (id, type, status, message, initiated, created_at, container_id, database, endpoint, backup_id, operation_id, paths, paths_to_exclude) VALUES ($id_1, $type_1, $status_1, $message_1, $initiated_1, $created_at_1, $container_id_1, $database_1, $endpoint_1, $backup_id_1, $operation_id_1, $paths_1, $paths_to_exclude_1)"} --- PASS: TestQueryBuilder_UpdateCreate (0.00s) === RUN TestQueryBuilder_CreateBackupSchedule 2024-09-23T20:45:10.631Z DEBUG queries/write.go:471 write query {"yql": "UPSERT INTO BackupSchedules (id, container_id, database, endpoint, name, active, crontab, ttl, paths, initiated, created_at, recovery_point_objective, last_backup_id, next_launch) VALUES ($id_0, $container_id_0, $database_0, $endpoint_0, $name_0, $active_0, $crontab_0, $ttl_0, $paths_0, $initiated_0, $created_at_0, $recovery_point_objective_0, $last_backup_id_0, $next_launch_0)"} --- PASS: TestQueryBuilder_CreateBackupSchedule (0.00s)
🟡 ydbcp/internal/connectors/s3	0	0	0	0ms
🧪 Tests (none) 🖨️ Output (none)
🟢 ydbcp/internal/handlers	24	0	0	1043ms
🧪 Tests 🟢TestBackupScheduleHandler 🟢TestDBOperationHandlerDeadlineExceededForRunningOperation 🟢TestDBOperationHandlerPendingOperationCompletedSuccessfully 🟢TestDBOperationHandlerRunningOperationCompletedSuccessfully 🟢TestRBOperationHandlerCancellingOperationCancelled 🟢TestRBOperationHandlerCancellingOperationCompletedSuccessfully 🟢TestRBOperationHandlerCancellingOperationInProgress 🟢TestRBOperationHandlerDeadlineExceededForCancellingOperation 🟢TestRBOperationHandlerDeadlineExceededForRunningOperation 🟢TestRBOperationHandlerInvalidOperationResponse 🟢TestRBOperationHandlerRetriableErrorForRunningOperation 🟢TestRBOperationHandlerRunningOperationCancelled 🟢TestRBOperationHandlerRunningOperationCompletedSuccessfully 🟢TestRBOperationHandlerRunningOperationInProgress 🟢TestTBOperationHandlerCancellingOperationCancelled 🟢TestTBOperationHandlerCancellingOperationCompletedSuccessfully 🟢TestTBOperationHandlerCancellingOperationInProgress 🟢TestTBOperationHandlerDeadlineExceededForCancellingOperation 🟢TestTBOperationHandlerDeadlineExceededForRunningOperation 🟢TestTBOperationHandlerInvalidOperationResponse 🟢TestTBOperationHandlerRetriableErrorForRunningOperation 🟢TestTBOperationHandlerRunningOperationCancelled 🟢TestTBOperationHandlerRunningOperationCompletedSuccessfully 🟢TestTBOperationHandlerRunningOperationInProgress 🖨️ Output === RUN TestDBOperationHandlerDeadlineExceededForRunningOperation 2024-09-24T10:05:21.574Z INFO handlers/delete_backup.go:38 DBOperationHandler {"OperationMessage": ""} --- PASS: TestDBOperationHandlerDeadlineExceededForRunningOperation (0.00s) === RUN TestDBOperationHandlerPendingOperationCompletedSuccessfully 2024-09-24T10:05:21.575Z INFO handlers/delete_backup.go:38 DBOperationHandler {"OperationMessage": ""} --- PASS: TestDBOperationHandlerPendingOperationCompletedSuccessfully (0.00s) === RUN TestDBOperationHandlerRunningOperationCompletedSuccessfully 2024-09-24T10:05:21.575Z INFO handlers/delete_backup.go:38 DBOperationHandler {"OperationMessage": ""} --- PASS: TestDBOperationHandlerRunningOperationCompletedSuccessfully (0.00s) === RUN TestRBOperationHandlerInvalidOperationResponse 2024-09-24T10:05:21.575Z INFO handlers/restore_backup.go:33 RBOperationHandler {"OperationMessage": ""} 2024-09-24T10:05:21.575Z INFO handlers/utils.go:48 getting operation status {"id": "07a6a4d9-6a8b-4b8f-89c0-d4296115f328", "type": "RB", "ydb_operation_id": "1"} --- PASS: TestRBOperationHandlerInvalidOperationResponse (0.00s) === RUN TestRBOperationHandlerDeadlineExceededForRunningOperation 2024-09-24T10:05:21.576Z INFO handlers/restore_backup.go:33 RBOperationHandler {"OperationMessage": ""} 2024-09-24T10:05:21.576Z INFO handlers/utils.go:48 getting operation status {"id": "309760ae-07da-4b7e-8fe6-3ed0430cfece", "type": "RB", "ydb_operation_id": "1"} --- PASS: TestRBOperationHandlerDeadlineExceededForRunningOperation (0.00s) === RUN TestRBOperationHandlerRunningOperationInProgress 2024-09-24T10:05:21.576Z INFO handlers/restore_backup.go:33 RBOperationHandler {"OperationMessage": ""} 2024-09-24T10:05:21.576Z INFO handlers/utils.go:48 getting operation status {"id": "85828ec5-44e9-4fe6-8211-73442b78ba61", "type": "RB", "ydb_operation_id": "1"} --- PASS: TestRBOperationHandlerRunningOperationInProgress (0.00s) === RUN TestRBOperationHandlerRunningOperationCompletedSuccessfully 2024-09-24T10:05:21.577Z INFO handlers/restore_backup.go:33 RBOperationHandler {"OperationMessage": ""} 2024-09-24T10:05:21.577Z INFO handlers/utils.go:48 getting operation status {"id": "4494ce96-b235-410c-8be6-4d71a7381133", "type": "RB", "ydb_operation_id": "1"} 2024-09-24T10:05:21.577Z INFO handlers/restore_backup.go:136 forgetting operation {"id": "4494ce96-b235-410c-8be6-4d71a7381133", "type": "RB", "ydb_operation_id": "1"} --- PASS: TestRBOperationHandlerRunningOperationCompletedSuccessfully (0.00s) === RUN TestRBOperationHandlerRunningOperationCancelled 2024-09-24T10:05:21.577Z INFO handlers/restore_backup.go:33 RBOperationHandler {"OperationMessage": ""} 2024-09-24T10:05:21.577Z INFO handlers/utils.go:48 getting operation status {"id": "53efb64a-b70e-4e39-a6ae-b790baf67dd0", "type": "RB", "ydb_operation_id": "1"} 2024-09-24T10:05:21.577Z INFO handlers/restore_backup.go:136 forgetting operation {"id": "53efb64a-b70e-4e39-a6ae-b790baf67dd0", "type": "RB", "ydb_operation_id": "1"} --- PASS: TestRBOperationHandlerRunningOperationCancelled (0.00s) === RUN TestRBOperationHandlerDeadlineExceededForCancellingOperation 2024-09-24T10:05:21.577Z INFO handlers/restore_backup.go:33 RBOperationHandler {"OperationMessage": ""} 2024-09-24T10:05:21.577Z INFO handlers/utils.go:48 getting operation status {"id": "e51fe409-22e0-4a7d-81e2-7d9554d898bb", "type": "RB", "ydb_operation_id": "1"} --- PASS: TestRBOperationHandlerDeadlineExceededForCancellingOperation (0.00s) === RUN TestRBOperationHandlerCancellingOperationInProgress 2024-09-24T10:05:21.578Z INFO handlers/restore_backup.go:33 RBOperationHandler {"OperationMessage": ""} 2024-09-24T10:05:21.578Z INFO handlers/utils.go:48 getting operation status {"id": "503e8156-67e2-469f-bb42-1df133a43c81", "type": "RB", "ydb_operation_id": "1"} --- PASS: TestRBOperationHandlerCancellingOperationInProgress (0.00s) === RUN TestRBOperationHandlerCancellingOperationCompletedSuccessfully 2024-09-24T10:05:21.578Z INFO handlers/restore_backup.go:33 RBOperationHandler {"OperationMessage": "operation was cancelled by user"} 2024-09-24T10:05:21.578Z INFO handlers/utils.go:48 getting operation status {"id": "541ffe2c-54fd-4675-8d17-40c76f981e5a", "type": "RB", "ydb_operation_id": "1"} 2024-09-24T10:05:21.578Z INFO handlers/restore_backup.go:136 forgetting operation {"id": "541ffe2c-54fd-4675-8d17-40c76f981e5a", "type": "RB", "ydb_operation_id": "1"} --- PASS: TestRBOperationHandlerCancellingOperationCompletedSuccessfully (0.00s) === RUN TestRBOperationHandlerCancellingOperationCancelled 2024-09-24T10:05:21.578Z INFO handlers/restore_backup.go:33 RBOperationHandler {"OperationMessage": ""} 2024-09-24T10:05:21.578Z INFO handlers/utils.go:48 getting operation status {"id": "4a64678f-cec3-449a-abe0-7d57aa4bbae7", "type": "RB", "ydb_operation_id": "1"} 2024-09-24T10:05:21.578Z INFO handlers/restore_backup.go:136 forgetting operation {"id": "4a64678f-cec3-449a-abe0-7d57aa4bbae7", "type": "RB", "ydb_operation_id": "1"} --- PASS: TestRBOperationHandlerCancellingOperationCancelled (0.00s) === RUN TestRBOperationHandlerRetriableErrorForRunningOperation 2024-09-24T10:05:21.579Z INFO handlers/restore_backup.go:33 RBOperationHandler {"OperationMessage": ""} 2024-09-24T10:05:21.579Z INFO handlers/utils.go:48 getting operation status {"id": "3e7c4a39-f2a8-4387-afbb-bbf68dd6244c", "type": "RB", "ydb_operation_id": "1"} 2024-09-24T10:05:21.579Z INFO handlers/utils.go:71 received retriable error {"id": "3e7c4a39-f2a8-4387-afbb-bbf68dd6244c", "type": "RB", "ydb_operation_id": "1"} --- PASS: TestRBOperationHandlerRetriableErrorForRunningOperation (0.00s) === RUN TestBackupScheduleHandler 2024-09-24T10:05:21.579Z INFO backup_operations/make_backup.go:145 Export operation started {"ClientDSN": "mydb.valid.commydb", "S3DestinationPrefix": "mydb/20240924_100521", "ClientOperationID": "f116f47c-ff93-4d3b-aa85-1bc7c546030e"} --- PASS: TestBackupScheduleHandler (0.00s) === RUN TestTBOperationHandlerInvalidOperationResponse 2024-09-24T10:05:21.580Z INFO handlers/take_backup.go:40 TBOperationHandler {"OperationMessage": ""} 2024-09-24T10:05:21.580Z INFO handlers/utils.go:48 getting operation status {"id": "2821e723-b655-4727-b791-1d67f20502f8", "type": "TB", "ydb_operation_id": "1"} --- PASS: TestTBOperationHandlerInvalidOperationResponse (0.00s) === RUN TestTBOperationHandlerDeadlineExceededForRunningOperation 2024-09-24T10:05:21.580Z INFO handlers/take_backup.go:40 TBOperationHandler {"OperationMessage": ""} 2024-09-24T10:05:21.580Z INFO handlers/utils.go:48 getting operation status {"id": "606cc74b-bc8c-4781-bdc6-44ee42682fb5", "type": "TB", "ydb_operation_id": "1"} --- PASS: TestTBOperationHandlerDeadlineExceededForRunningOperation (0.00s) === RUN TestTBOperationHandlerRunningOperationInProgress 2024-09-24T10:05:21.580Z INFO handlers/take_backup.go:40 TBOperationHandler {"OperationMessage": ""} 2024-09-24T10:05:21.580Z INFO handlers/utils.go:48 getting operation status {"id": "4c4d176d-c96f-4826-b7eb-15a4317c5ce4", "type": "TB", "ydb_operation_id": "1"} --- PASS: TestTBOperationHandlerRunningOperationInProgress (0.00s) === RUN TestTBOperationHandlerRunningOperationCompletedSuccessfully 2024-09-24T10:05:21.581Z INFO handlers/take_backup.go:40 TBOperationHandler {"OperationMessage": ""} 2024-09-24T10:05:21.581Z INFO handlers/utils.go:48 getting operation status {"id": "28cd327b-e897-4096-8bc7-1effa47cab2e", "type": "TB", "ydb_operation_id": "1"} --- PASS: TestTBOperationHandlerRunningOperationCompletedSuccessfully (0.00s) === RUN TestTBOperationHandlerRunningOperationCancelled 2024-09-24T10:05:21.581Z INFO handlers/take_backup.go:40 TBOperationHandler {"OperationMessage": ""} 2024-09-24T10:05:21.581Z INFO handlers/utils.go:48 getting operation status {"id": "77224985-1e54-419e-899f-578a4a129805", "type": "TB", "ydb_operation_id": "1"} --- PASS: TestTBOperationHandlerRunningOperationCancelled (0.00s) === RUN TestTBOperationHandlerDeadlineExceededForCancellingOperation 2024-09-24T10:05:21.581Z INFO handlers/take_backup.go:40 TBOperationHandler {"OperationMessage": ""} 2024-09-24T10:05:21.581Z INFO handlers/utils.go:48 getting operation status {"id": "4c2135e2-8dac-4216-a237-34db5fe53a42", "type": "TB", "ydb_operation_id": "1"} --- PASS: TestTBOperationHandlerDeadlineExceededForCancellingOperation (0.00s) === RUN TestTBOperationHandlerCancellingOperationInProgress 2024-09-24T10:05:21.582Z INFO handlers/take_backup.go:40 TBOperationHandler {"OperationMessage": ""} 2024-09-24T10:05:21.582Z INFO handlers/utils.go:48 getting operation status {"id": "116f2e3d-67da-44f5-945d-4c9725b3a507", "type": "TB", "ydb_operation_id": "1"} --- PASS: TestTBOperationHandlerCancellingOperationInProgress (0.00s) === RUN TestTBOperationHandlerCancellingOperationCompletedSuccessfully 2024-09-24T10:05:21.582Z INFO handlers/take_backup.go:40 TBOperationHandler {"OperationMessage": "operation was cancelled by user"} 2024-09-24T10:05:21.582Z INFO handlers/utils.go:48 getting operation status {"id": "ff667704-2f9e-460a-b699-7fbc18165c01", "type": "TB", "ydb_operation_id": "1"} Operation was completed despite cancellation: operation was cancelled by user--- PASS: TestTBOperationHandlerCancellingOperationCompletedSuccessfully (0.00s) === RUN TestTBOperationHandlerCancellingOperationCancelled 2024-09-24T10:05:21.582Z INFO handlers/take_backup.go:40 TBOperationHandler {"OperationMessage": ""} 2024-09-24T10:05:21.582Z INFO handlers/utils.go:48 getting operation status {"id": "0c0468f7-d72b-4ed8-a4f4-c39b3fe230d4", "type": "TB", "ydb_operation_id": "1"} --- PASS: TestTBOperationHandlerCancellingOperationCancelled (0.00s) === RUN TestTBOperationHandlerRetriableErrorForRunningOperation 2024-09-24T10:05:21.582Z INFO handlers/take_backup.go:40 TBOperationHandler {"OperationMessage": ""} 2024-09-24T10:05:21.583Z INFO handlers/utils.go:48 getting operation status {"id": "365933ae-d3e6-44a0-8b4b-f252ef97d5c8", "type": "TB", "ydb_operation_id": "1"} 2024-09-24T10:05:21.583Z INFO handlers/utils.go:71 received retriable error {"id": "365933ae-d3e6-44a0-8b4b-f252ef97d5c8", "type": "TB", "ydb_operation_id": "1"} --- PASS: TestTBOperationHandlerRetriableErrorForRunningOperation (0.00s)
🟢 ydbcp/internal/processor	2	0	0	1024ms
🧪 Tests 🟢TestOperationHandlerRegistry 🟢TestProcessor 🖨️ Output === RUN TestProcessor 2024-09-24T10:05:21.563Z DEBUG processor/processor.go:76 Operation Processor started {"period": "10s"} 2024-09-24T10:05:21.563Z DEBUG processor/processor.go:134 operation already running {"RunID": "460fc004-bd60-4207-bfb3-ca2f1e28f1bc", "OperationID": "3ddd05d3-4403-45bf-ae22-b5c0f8d39e4f", "OperationType": "TB", "OperationState": "PENDING", "operation": "Operation, id 3ddd05d3-4403-45bf-ae22-b5c0f8d39e4f, type TB, state PENDING"} 2024-09-24T10:05:21.563Z DEBUG processor/processor.go:134 operation already running {"RunID": "da251167-455f-4e5c-b803-8d35de3c49b4", "OperationID": "3ddd05d3-4403-45bf-ae22-b5c0f8d39e4f", "OperationType": "TB", "OperationState": "PENDING", "operation": "Operation, id 3ddd05d3-4403-45bf-ae22-b5c0f8d39e4f, type TB, state PENDING"} 2024-09-24T10:05:21.563Z DEBUG processor/processor.go:147 start operation handler {"RunID": "17a26b86-f89e-4a9a-92ea-ca48a6a0cc27", "OperationID": "3ddd05d3-4403-45bf-ae22-b5c0f8d39e4f", "OperationType": "TB", "OperationState": "PENDING", "operation": "Operation, id 3ddd05d3-4403-45bf-ae22-b5c0f8d39e4f, type TB, state PENDING"} 2024-09-24T10:05:21.563Z DEBUG processor/processor_test.go:41 TB handler called for operation {"RunID": "17a26b86-f89e-4a9a-92ea-ca48a6a0cc27", "OperationID": "3ddd05d3-4403-45bf-ae22-b5c0f8d39e4f", "OperationType": "TB", "OperationState": "PENDING", "operation": "Operation, id 3ddd05d3-4403-45bf-ae22-b5c0f8d39e4f, type TB, state PENDING"} 2024-09-24T10:05:21.564Z DEBUG processor/processor.go:159 operation handler finished successfully {"RunID": "17a26b86-f89e-4a9a-92ea-ca48a6a0cc27", "OperationID": "3ddd05d3-4403-45bf-ae22-b5c0f8d39e4f", "OperationType": "TB", "OperationState": "PENDING", "operation": "Operation, id 3ddd05d3-4403-45bf-ae22-b5c0f8d39e4f, type TB, state DONE"} 2024-09-24T10:05:21.564Z DEBUG processor/processor.go:175 operation handler is marked as finished {"OperationID": "3ddd05d3-4403-45bf-ae22-b5c0f8d39e4f"} 2024-09-24T10:05:21.564Z DEBUG processor/processor.go:81 Operation Processor stopping. Waiting for running operations... 2024-09-24T10:05:21.564Z DEBUG processor/processor.go:84 Operation Processor stopped --- PASS: TestProcessor (0.00s) === RUN TestOperationHandlerRegistry --- PASS: TestOperationHandlerRegistry (0.00s)
🟡 ydbcp/internal/server	0	0	0	0ms
🧪 Tests (none) 🖨️ Output (none)
🟡 ydbcp/internal/server/grpcinfo	0	0	0	0ms
🧪 Tests (none) 🖨️ Output (none)
🟡 ydbcp/internal/server/services/backup	0	0	0	0ms
🧪 Tests (none) 🖨️ Output (none)
🟡 ydbcp/internal/server/services/backup_schedule	0	0	0	0ms
🧪 Tests (none) 🖨️ Output (none)
🟡 ydbcp/internal/server/services/operation	0	0	0	0ms
🧪 Tests (none) 🖨️ Output (none)
🟢 ydbcp/internal/ttl_watcher	0	0	0	1018ms
🧪 Tests (none) 🖨️ Output (none)
🟡 ydbcp/internal/types	0	0	0	0ms
🧪 Tests (none) 🖨️ Output (none)
🟡 ydbcp/internal/util/ticker	0	0	0	0ms
🧪 Tests (none) 🖨️ Output (none)
🟡 ydbcp/internal/util/timer	0	0	0	0ms
🧪 Tests (none) 🖨️ Output (none)
🟡 ydbcp/internal/util/xlog	0	0	0	0ms
🧪 Tests (none) 🖨️ Output (none)
🟡 ydbcp/pkg/plugins/auth	0	0	0	0ms
🧪 Tests (none) 🖨️ Output (none)
🟡 ydbcp/pkg/proto/ydbcp/v1alpha1	0	0	0	0ms
🧪 Tests (none) 🖨️ Output (none)
🟡 ydbcp/plugins/auth_nebius	0	0	0	0ms
🧪 Tests (none) 🖨️ Output (none)
🟡 ydbcp/plugins/auth_nebius/proto/iam/v1	0	0	0	0ms
🧪 Tests (none) 🖨️ Output (none)

[qrort]

maybe do some cosmetic changes, lgtm