Skip to content

Commit

Permalink
[Fix][mysql-cdc] Fix GTIDs on startup to correctly recover from check…
Browse files Browse the repository at this point in the history 
…point (#8528)
  • Loading branch information
@jw-itq
jw-itq authored Feb 7, 2025
1 parent aadfe99 commit 82e4096
Show file tree
Hide file tree
Showing 15 changed files with 2,469 additions and 34 deletions.
22 changes: 11 additions & 11 deletions docs/en/connector-v2/sink/Doris.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -49,17 +49,17 @@ The internal implementation of Doris sink connector is cached and imported by st
| sink.label-prefix | String | Yes | - | The label prefix used by stream load imports. In the 2pc scenario, global uniqueness is required to ensure the EOS semantics of SeaTunnel. |
| sink.enable-2pc | bool | No | false | Whether to enable two-phase commit (2pc), the default is false. For two-phase commit, please refer to [here](https://doris.apache.org/docs/data-operate/transaction?_highlight=two&_highlight=phase#stream-load-2pc). |
| sink.enable-delete | bool | No | - | Whether to enable deletion. This option requires Doris table to enable batch delete function (0.15+ version is enabled by default), and only supports Unique model. you can get more detail at this [link](https://doris.apache.org/docs/dev/data-operate/delete/batch-delete-manual/) |
| sink.check-interval | int | No | 10000 | check exception with the interval while loading |
| sink.max-retries | int | No | 3 | the max retry times if writing records to database failed |
| sink.buffer-size | int | No | 256 * 1024 | the buffer size to cache data for stream load. |
| sink.buffer-count | int | No | 3 | the buffer count to cache data for stream load. |
| doris.batch.size | int | No | 1024 | the batch size of the write to doris each http request, when the row reaches the size or checkpoint is executed, the data of cached will write to server. |
| needs_unsupported_type_casting | boolean | No | false | Whether to enable the unsupported type casting, such as Decimal64 to Double |
| schema_save_mode | Enum | no | CREATE_SCHEMA_WHEN_NOT_EXIST | the schema save mode, please refer to `schema_save_mode` below |
| data_save_mode | Enum | no | APPEND_DATA | the data save mode, please refer to `data_save_mode` below |
| save_mode_create_template | string | no | see below | see below |
| custom_sql | String | no | - | When data_save_mode selects CUSTOM_PROCESSING, you should fill in the CUSTOM_SQL parameter. This parameter usually fills in a SQL that can be executed. SQL will be executed before synchronization tasks. |
| doris.config | map | yes | - | This option is used to support operations such as `insert`, `delete`, and `update` when automatically generate sql,and supported formats. |
| sink.check-interval | int | No | 10000 | check exception with the interval while loading |
| sink.max-retries | int | No | 3 | the max retry times if writing records to database failed |
| sink.buffer-size | int | No | 256 * 1024 | the buffer size to cache data for stream load. |
| sink.buffer-count | int | No | 3 | the buffer count to cache data for stream load. |
| doris.batch.size | int | No | 1024 | the batch size of the write to doris each http request, when the row reaches the size or checkpoint is executed, the data of cached will write to server. |
| needs_unsupported_type_casting | boolean | No | false | Whether to enable the unsupported type casting, such as Decimal64 to Double |
| schema_save_mode | Enum | no | CREATE_SCHEMA_WHEN_NOT_EXIST | the schema save mode, please refer to `schema_save_mode` below |
| data_save_mode | Enum | no | APPEND_DATA | the data save mode, please refer to `data_save_mode` below |
| save_mode_create_template | string | no | see below | see below |
| custom_sql | String | no | - | When data_save_mode selects CUSTOM_PROCESSING, you should fill in the CUSTOM_SQL parameter. This parameter usually fills in a SQL that can be executed. SQL will be executed before synchronization tasks. |
| doris.config | map | yes | - | This option is used to support operations such as `insert`, `delete`, and `update` when automatically generate sql,and supported formats. |

### schema_save_mode[Enum]

Expand Down
22 changes: 11 additions & 11 deletions docs/zh/connector-v2/sink/Doris.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -48,17 +48,17 @@ Doris Sink连接器的内部实现是通过stream load批量缓存和导入的
| sink.label-prefix | String | Yes | - | stream load导入使用的标签前缀。 在2pc场景下，需要全局唯一性来保证SeaTunnel的EOS语义。 |
| sink.enable-2pc | bool | No | false | 是否启用两阶段提交（2pc），默认为 false。 对于两阶段提交，请参考[此处](https://doris.apache.org/docs/data-operate/transaction?_highlight=two&_highlight=phase#stream-load-2pc)。 |
| sink.enable-delete | bool | No | - | 是否启用删除。 该选项需要Doris表开启批量删除功能（0.15+版本默认开启），且仅支持Unique模型。 您可以在此[link](https://doris.apache.org/docs/dev/data-operate/delete/batch-delete-manual/)获得更多详细信息 |
| sink.check-interval | int | No | 10000 | 加载过程中检查异常时间间隔。 |
| sink.max-retries | int | No | 3 | 向数据库写入记录失败时的最大重试次数。 |
| sink.buffer-size | int | No | 256 * 1024 | 用于缓存stream load数据的缓冲区大小。 |
| sink.buffer-count | int | No | 3 | 用于缓存stream load数据的缓冲区计数。 |
| doris.batch.size | int | No | 1024 | 每次http请求写入doris的批量大小，当row达到该大小或者执行checkpoint时，缓存的数据就会写入服务器。 |
| needs_unsupported_type_casting | boolean | No | false | 是否启用不支持的类型转换，例如 Decimal64 到 Double。 |
| schema_save_mode | Enum | no | CREATE_SCHEMA_WHEN_NOT_EXIST | schema保存模式，请参考下面的`schema_save_mode` |
| data_save_mode | Enum | no | APPEND_DATA | 数据保存模式，请参考下面的`data_save_mode`。 |
| save_mode_create_template | string | no | see below | 见下文。 |
| custom_sql | String | no | - | 当data_save_mode选择CUSTOM_PROCESSING时，需要填写CUSTOM_SQL参数。 该参数通常填写一条可以执行的SQL。 SQL将在同步任务之前执行。 |
| doris.config | map | yes | - | 该选项用于支持自动生成sql时的insert、delete、update等操作，以及支持的格式。 |
| sink.check-interval | int | No | 10000 | 加载过程中检查异常时间间隔。 |
| sink.max-retries | int | No | 3 | 向数据库写入记录失败时的最大重试次数。 |
| sink.buffer-size | int | No | 256 * 1024 | 用于缓存stream load数据的缓冲区大小。 |
| sink.buffer-count | int | No | 3 | 用于缓存stream load数据的缓冲区计数。 |
| doris.batch.size | int | No | 1024 | 每次http请求写入doris的批量大小，当row达到该大小或者执行checkpoint时，缓存的数据就会写入服务器。 |
| needs_unsupported_type_casting | boolean | No | false | 是否启用不支持的类型转换，例如 Decimal64 到 Double。 |
| schema_save_mode | Enum | no | CREATE_SCHEMA_WHEN_NOT_EXIST | schema保存模式，请参考下面的`schema_save_mode` |
| data_save_mode | Enum | no | APPEND_DATA | 数据保存模式，请参考下面的`data_save_mode`。 |
| save_mode_create_template | string | no | see below | 见下文。 |
| custom_sql | String | no | - | 当data_save_mode选择CUSTOM_PROCESSING时，需要填写CUSTOM_SQL参数。 该参数通常填写一条可以执行的SQL。 SQL将在同步任务之前执行。 |
| doris.config | map | yes | - | 该选项用于支持自动生成sql时的insert、delete、update等操作，以及支持的格式。 |

### schema_save_mode[Enum]

Expand Down
Loading

0 comments on commit 82e4096

Please sign in to comment.