From 63beef70eedf3b4ca3ebeb1fca7fab139ea6c5cb Mon Sep 17 00:00:00 2001 From: hillium Date: Tue, 4 Mar 2025 13:50:41 +0800 Subject: [PATCH 1/4] refined br-log-arch --- br/br-log-architecture.md | 58 +++++++++++++++++++++------------------ 1 file changed, 32 insertions(+), 26 deletions(-) diff --git a/br/br-log-architecture.md b/br/br-log-architecture.md index fe3f9e25b3810..c5f11d3ff5ee8 100644 --- a/br/br-log-architecture.md +++ b/br/br-log-architecture.md @@ -108,18 +108,24 @@ Log backup generates the following types of files: . ├── v1 │   ├── backupmeta -│   │   ├── {min_restored_ts}-{uuid}.meta -│   │   ├── {checkpoint}-{uuid}.meta +│   │   ├── ... +│   │   └── {resolved_ts}-{uuid}.meta │   ├── global_checkpoint -│   │   ├── {store_id}.ts -│   ├── {date} -│   │   ├── {hour} -│   │   │   ├── {store_id} -│   │   │   │   ├── {min_ts}-{uuid}.log -│   │   │   │   ├── {min_ts}-{uuid}.log -├── v1_stream_truncate_safepoint.txt +│   │   └── {store_id}.ts +│   └── {date} +│      └── {hour} +│         └── {store_id} +│            ├── ... +│            └── {min_ts}-{uuid}.log +└── v1_stream_truncate_safepoint.txt ``` +Among them: + +- `backupmeta` contains the metadata of the backup. The `resolved_ts` in the file name indicates the progress of the backup: data before this TSO has been fully backed up. However, please note that this data only reflects the progress of some shards, so this number is not very meaningful to users. +- `global_checkpoint` stores the global progress of the backup: it records the latest point in time that can be restored to using `br restore point`. +- The `{date}/{hour}` directory stores the backup data for the corresponding date. Please note that when cleaning up storage, you should use `br log truncate` instead of manually deleting the data, because the metadata points to this data though deleted manually, and manually deleting it will cause problems. + The following is an example: ``` @@ -129,24 +135,24 @@ The following is an example: │   │   ├── ... │   │   ├── 435213818858112001-e2569bda-a75a-4411-88de-f469b49d6256.meta │   │   ├── 435214043785779202-1780f291-3b8a-455e-a31d-8a1302c43ead.meta -│   │   ├── 435214443785779202-224f1408-fff5-445f-8e41-ca4fcfbd2a67.meta +│   │   └── 435214443785779202-224f1408-fff5-445f-8e41-ca4fcfbd2a67.meta │   ├── global_checkpoint │   │   ├── 1.ts │   │   ├── 2.ts -│   │   ├── 3.ts -│   ├── 20220811 -│   │   ├── 03 -│   │   │   ├── 1 -│   │   │   │   ├── ... -│   │   │   │   ├── 435213866703257604-60fcbdb6-8f55-4098-b3e7-2ce604dafe54.log -│   │   │   │   ├── 435214023989657606-72ce65ff-1fa8-4705-9fd9-cb4a1e803a56.log -│   │   │   ├── 2 -│   │   │   │   ├── ... -│   │   │   │   ├── 435214102632857605-11deba64-beff-4414-bc9c-7a161b6fb22c.log -│   │   │   │   ├── 435214417205657604-e6980303-cbaa-4629-a863-1e745d7b8aed.log -│   │   │   ├── 3 -│   │   │   │   ├── ... -│   │   │   │   ├── 435214495848857605-7bf65e92-8c43-427e-b81e-f0050bd40be0.log -│   │   │   │   ├── 435214574492057604-80d3b15e-3d9f-4b0c-b133-87ed3f6b2697.log -├── v1_stream_truncate_safepoint.txt +│   │   └── 3.ts +│   └── 20220811 +│      └── 03 +│         ├── 1 +│         │   ├── ... +│         │   ├── 435213866703257604-60fcbdb6-8f55-4098-b3e7-2ce604dafe54.log +│         │   └── 435214023989657606-72ce65ff-1fa8-4705-9fd9-cb4a1e803a56.log +│         ├── 2 +│         │   ├── ... +│         │   ├── 435214102632857605-11deba64-beff-4414-bc9c-7a161b6fb22c.log +│         │   └── 435214417205657604-e6980303-cbaa-4629-a863-1e745d7b8aed.log +│         └── 3 +│            ├── ... +│            ├── 435214495848857605-7bf65e92-8c43-427e-b81e-f0050bd40be0.log +│            └── 435214574492057604-80d3b15e-3d9f-4b0c-b133-87ed3f6b2697.log +└── v1_stream_truncate_safepoint.txt ``` From c06455624a3fc7d18705658ff15aca9ce8ce48a5 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E5=B1=B1=E5=B2=9A?= <36239017+YuJuncen@users.noreply.github.com> Date: Tue, 4 Mar 2025 17:17:42 +0800 Subject: [PATCH 2/4] Apply suggestions from code review Co-authored-by: Lilian Lee --- br/br-log-architecture.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/br/br-log-architecture.md b/br/br-log-architecture.md index c5f11d3ff5ee8..88483f580ec57 100644 --- a/br/br-log-architecture.md +++ b/br/br-log-architecture.md @@ -120,11 +120,11 @@ Log backup generates the following types of files: └── v1_stream_truncate_safepoint.txt ``` -Among them: +Explanation of the backup file directory structure: -- `backupmeta` contains the metadata of the backup. The `resolved_ts` in the file name indicates the progress of the backup: data before this TSO has been fully backed up. However, please note that this data only reflects the progress of some shards, so this number is not very meaningful to users. -- `global_checkpoint` stores the global progress of the backup: it records the latest point in time that can be restored to using `br restore point`. -- The `{date}/{hour}` directory stores the backup data for the corresponding date. Please note that when cleaning up storage, you should use `br log truncate` instead of manually deleting the data, because the metadata points to this data though deleted manually, and manually deleting it will cause problems. +- `backupmeta`: contains backup metadata. The `resolved_ts` in the filename indicates the backup progress, meaning that data before this TSO has been fully backed up. However, note that this TSO only reflects the progress of certain shards, so its significance is limited. +- `global_checkpoint`: represents the global backup progress. It records the latest point in time to which data can be restored using `br restore point`. +- `{date}/{hour}`: stores backup data for the corresponding date and hour. When cleaning up storage, always use `br log truncate` instead of manually deleting data. This is because the metadata references the data in this directory, and manual deletion might lead to restore failures or data inconsistencies after restore. The following is an example: From a49c05a244df4e8de730c4c8c5d4bf2a0b97c850 Mon Sep 17 00:00:00 2001 From: Lilian Lee Date: Tue, 4 Mar 2025 17:52:19 +0800 Subject: [PATCH 3/4] Apply suggestions from code review Co-authored-by: BornChanger <97348524+BornChanger@users.noreply.github.com> --- br/br-log-architecture.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/br/br-log-architecture.md b/br/br-log-architecture.md index 88483f580ec57..64ffbbdbb4a18 100644 --- a/br/br-log-architecture.md +++ b/br/br-log-architecture.md @@ -122,7 +122,7 @@ Log backup generates the following types of files: Explanation of the backup file directory structure: -- `backupmeta`: contains backup metadata. The `resolved_ts` in the filename indicates the backup progress, meaning that data before this TSO has been fully backed up. However, note that this TSO only reflects the progress of certain shards, so its significance is limited. +- `backupmeta`: contains backup metadata. The `resolved_ts` in the filename indicates the backup progress, meaning that data before this TSO has been fully backed up. However, note that this TSO only reflects the progress of certain shards. - `global_checkpoint`: represents the global backup progress. It records the latest point in time to which data can be restored using `br restore point`. - `{date}/{hour}`: stores backup data for the corresponding date and hour. When cleaning up storage, always use `br log truncate` instead of manually deleting data. This is because the metadata references the data in this directory, and manual deletion might lead to restore failures or data inconsistencies after restore. From 54508bef4a9c054d37bf12b7fa45ffbefc19c3ee Mon Sep 17 00:00:00 2001 From: Lilian Lee Date: Tue, 4 Mar 2025 18:16:19 +0800 Subject: [PATCH 4/4] Apply suggestions from code review Co-authored-by: Grace Cai --- br/br-log-architecture.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/br/br-log-architecture.md b/br/br-log-architecture.md index 64ffbbdbb4a18..c3c7a38ab44fb 100644 --- a/br/br-log-architecture.md +++ b/br/br-log-architecture.md @@ -122,7 +122,7 @@ Log backup generates the following types of files: Explanation of the backup file directory structure: -- `backupmeta`: contains backup metadata. The `resolved_ts` in the filename indicates the backup progress, meaning that data before this TSO has been fully backed up. However, note that this TSO only reflects the progress of certain shards. +- `backupmeta`: stores backup metadata. The `resolved_ts` in the filename indicates the backup progress, meaning that data before this TSO has been fully backed up. However, note that this TSO only reflects the progress of certain shards. - `global_checkpoint`: represents the global backup progress. It records the latest point in time to which data can be restored using `br restore point`. - `{date}/{hour}`: stores backup data for the corresponding date and hour. When cleaning up storage, always use `br log truncate` instead of manually deleting data. This is because the metadata references the data in this directory, and manual deletion might lead to restore failures or data inconsistencies after restore.