chore(deps): Update ClickHouse SQL driver to v2 #16423

AndreKR · 2025-01-22T11:27:44Z

Summary

This updates the clickhouse SQL driver (and its dependencies) to v2.

This is mainly because the v1 driver doesn't support the HTTP protocol, but it also helps with some issues regarding DSN parsing that v1 suffers from.

This includes another commit that changes the default TableTemplate so that it works with ClickHouse. Previously everyone who wanted to use ClickHouse had to override TableTemplate.

Checklist

No AI generated code was used in this PR

Related issues

resolves #16422

srebhan · 2025-01-28T12:49:38Z

@AndreKR do current v1 DSNs work with the new driver?

AndreKR · 2025-01-28T13:41:19Z

No, I don't think so. The main difference is that in v1 database was a parameter (tcp://.../?database=<database>) while in v2 it's the path (tcp://.../<database>?).

srebhan · 2025-01-29T11:59:47Z

Do you see some way of converting old DSNs? Maybe DSNs can be converted by parsing them with the v1 parser and (if successful) then create a v2 DSN from it? I'm asking because Telegraf cannot do these kind of breaking changes, therefore we need to find a way to keep the old configs (and thus DSNs) working...

AndreKR · 2025-01-29T12:05:47Z

Yes, I think that would be possible. I'll prepare something.

AndreKR · 2025-01-31T03:42:44Z

@srebhan I have added code that converts parameters in the DSN that have been renamed or have a different format to their v2 equivalents. A warning is logged for each legacy parameter. A number of parameters (tls_config, no_delay, write_timeout, block_size, check_connection_liveness) have no equivalent in v2. These are simply removed from the DSN and a warning is logged as well.

srebhan

Thanks @AndreKR for the amazing work! I have some comments in the code...

Additionally, did you test that the conversion function leaves newer v2 DSNs unchanged? Would be good to get some testing there...

Last but not least, please also add a note in the Important Changes section of the Changelog about the change, something like

- PR [#16423](https://github.com/influxdata/telegraf/pull/16423) converts the clickhouse drivers to the v2 version.
  This new version also requires a [new format for the DSN](url). The plugin tries
  its best to convert the old DSN to the new format but might not be able to do so.
  Please check for warnings in your log file and convert to the new format as soon
  as possible

srebhan · 2025-01-31T12:21:03Z

plugins/outputs/sql/sample.conf

@@ -18,6 +18,7 @@
  ##  {TABLE} - table name as a quoted identifier
  ##  {TABLELITERAL} - table name as a quoted string literal
  ##  {COLUMNS} - column definitions (list of quoted identifiers and types)
+  ##  {TAG_COLUMNS} - tag column definitions (list of quoted identifiers)


Please remove this from this PR as this is an unrelated feature!

Moving this to a separate PR.

Please note that without this fix, the SQL output plugin simply does not work with the recommended ClickHouse config, neither with v1 nor with v2. The only reason the existing test doesn't catch this is because the test overrides the create table template.

plugins/outputs/sql/sql.go

srebhan · 2025-01-31T12:28:44Z

plugins/outputs/sql/README.md

-format](https://github.com/ClickHouse/clickhouse-go/tree/v1.5.4#dsn) than its
-newer `v2.*` version.
+Note that even when the DSN is specified as `https://` the `secure=true`
+parameter is still required.  



How about adding a warning here that the DSN now should adhere to the v2 schema including a link to how a DSN might be constructed? Furthermore, add a brief note about the fact that

The plugin tries to convert v1 DSNs to v2 but as both schemata are not fully equivalent some parameters might not work anymore. Please check for warnings in your log file and refer to the DSN v2 documentation for available options

srebhan · 2025-01-31T12:30:29Z

plugins/outputs/sql/sql_test.go

@@ -353,7 +353,6 @@ func TestClickHouseIntegration(t *testing.T) {
 		WaitingFor: wait.ForAll(
 			wait.NewHTTPStrategy("/").WithPort(nat.Port("8123")),
 			wait.ForListeningPort(nat.Port(servicePort)),
-			wait.ForLog("Saved preprocessed configuration to '/var/lib/clickhouse/preprocessed_configs/users.xml'").WithOccurrence(2),


Are you absolutely sure the server is fully up, including data being inserted to the database and the server is accepting queries? If not, this can be racy and we end up with flaky tests!

So I took another look and indeed it looks like opening the ports is the very last step in the initialization, long after init.sql has run. We could theoretically wait for the corresponding log output...

2025.02.01 23:37:37.879571 [ 1 ] {} <Information> Application: Listening for http://0.0.0.0:8123 2025.02.01 23:37:37.880204 [ 1 ] {} <Information> Application: Listening for native protocol (tcp): 0.0.0.0:9000 2025.02.01 23:37:37.880607 [ 1 ] {} <Information> Application: Listening for MySQL compatibility protocol: 0.0.0.0:9004 2025.02.01 23:37:37.881120 [ 1 ] {} <Information> Application: Listening for PostgreSQL compatibility protocol: 0.0.0.0:9005 2025.02.01 23:37:37.881492 [ 1 ] {} <Information> Application: Ready for connections.

But since this would be completely equivalent to just waiting for the port to be open, I don't think it would be useful.

This updates the clickhouse SQL driver for the SQL output plugin to v2. This keeps some compatibility with v1 DSNs by converting parameters from the v1 to the v2 format while logging a warning. Some parameters are no longer supported, those are removed while logging a warning as well.

AndreKR · 2025-02-02T00:19:09Z

Additionally, did you test that the conversion function leaves newer v2 DSNs unchanged? Would be good to get some testing there...

I had a test about leaving a v2 URL untouched, I have added another one for a v2 URL containing parameters that would be changed if they had incompatible values but they have already compatible values.

Last but not least, please also add a note in the Important Changes section of the Changelog about the change

Done.

telegraf-tiger · 2025-02-02T00:33:29Z

Download PR build artifacts for linux_amd64.tar.gz, darwin_arm64.tar.gz, and windows_amd64.zip.
Downloads for additional architectures and packages are available below.

⚠️ This pull request increases the Telegraf binary size by 1.20 % for linux amd64 (new size: 287.4 MB, nightly size 284.0 MB)

📦 Click here to get additional PR build artifacts

Artifact URLs

DEB	RPM	TAR GZ	ZIP
amd64.deb	aarch64.rpm	darwin_amd64.tar.gz	windows_amd64.zip
arm64.deb	armel.rpm	darwin_arm64.tar.gz	windows_arm64.zip
armel.deb	armv6hl.rpm	freebsd_amd64.tar.gz	windows_i386.zip
armhf.deb	i386.rpm	freebsd_armv7.tar.gz
i386.deb	ppc64le.rpm	freebsd_i386.tar.gz
mips.deb	riscv64.rpm	linux_amd64.tar.gz
mipsel.deb	s390x.rpm	linux_arm64.tar.gz
ppc64el.deb	x86_64.rpm	linux_armel.tar.gz
riscv64.deb		linux_armhf.tar.gz
s390x.deb		linux_i386.tar.gz
		linux_mips.tar.gz
		linux_mipsel.tar.gz
		linux_ppc64le.tar.gz
		linux_riscv64.tar.gz
		linux_s390x.tar.gz

AndreKR force-pushed the clickhouse-upgrade branch 2 times, most recently from f1c28d0 to f797456 Compare January 22, 2025 11:47

AndreKR marked this pull request as draft January 22, 2025 11:51

AndreKR changed the title ~~Update ClickHouse driver~~ chore(deps): Update ClickHouse SQL driver to v2 Jan 22, 2025

telegraf-tiger bot added the chore label Jan 22, 2025

AndreKR force-pushed the clickhouse-upgrade branch 6 times, most recently from 80069ac to d6a09dd Compare January 26, 2025 21:21

AndreKR marked this pull request as ready for review January 26, 2025 21:41

AndreKR force-pushed the clickhouse-upgrade branch from d6a09dd to 3c523dd Compare January 26, 2025 22:11

AndreKR marked this pull request as draft January 26, 2025 22:39

AndreKR force-pushed the clickhouse-upgrade branch 2 times, most recently from 5cd941f to d2682fd Compare January 28, 2025 01:24

AndreKR marked this pull request as ready for review January 28, 2025 11:59

AndreKR force-pushed the clickhouse-upgrade branch from a5cd7e3 to d2682fd Compare January 28, 2025 12:30

srebhan self-assigned this Jan 28, 2025

srebhan added the dependencies Pull requests that update a dependency file label Jan 28, 2025

AndreKR marked this pull request as draft January 29, 2025 12:05

AndreKR force-pushed the clickhouse-upgrade branch 4 times, most recently from 58bd017 to 88fce4f Compare January 31, 2025 03:09

AndreKR marked this pull request as ready for review January 31, 2025 03:42

srebhan reviewed Jan 31, 2025

View reviewed changes

AndreKR marked this pull request as draft February 1, 2025 22:41

AndreKR force-pushed the clickhouse-upgrade branch 2 times, most recently from b686961 to 6aa4837 Compare February 1, 2025 23:56

AndreKR force-pushed the clickhouse-upgrade branch from 6aa4837 to d73a52e Compare February 2, 2025 00:08

AndreKR marked this pull request as ready for review February 2, 2025 00:19

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

chore(deps): Update ClickHouse SQL driver to v2 #16423

chore(deps): Update ClickHouse SQL driver to v2 #16423

AndreKR commented Jan 22, 2025

srebhan commented Jan 28, 2025

AndreKR commented Jan 28, 2025

srebhan commented Jan 29, 2025

AndreKR commented Jan 29, 2025

AndreKR commented Jan 31, 2025

srebhan left a comment

srebhan Jan 31, 2025

AndreKR Feb 1, 2025

srebhan Jan 31, 2025

AndreKR Feb 2, 2025

srebhan Jan 31, 2025

AndreKR Feb 1, 2025

AndreKR commented Feb 2, 2025

telegraf-tiger bot commented Feb 2, 2025

Artifact URLs

chore(deps): Update ClickHouse SQL driver to v2 #16423

Are you sure you want to change the base?

chore(deps): Update ClickHouse SQL driver to v2 #16423

Conversation

AndreKR commented Jan 22, 2025

Summary

Checklist

Related issues

srebhan commented Jan 28, 2025

AndreKR commented Jan 28, 2025

srebhan commented Jan 29, 2025

AndreKR commented Jan 29, 2025

AndreKR commented Jan 31, 2025

srebhan left a comment

Choose a reason for hiding this comment

srebhan Jan 31, 2025

Choose a reason for hiding this comment

AndreKR Feb 1, 2025

Choose a reason for hiding this comment

srebhan Jan 31, 2025

Choose a reason for hiding this comment

AndreKR Feb 2, 2025

Choose a reason for hiding this comment

srebhan Jan 31, 2025

Choose a reason for hiding this comment

AndreKR Feb 1, 2025

Choose a reason for hiding this comment

AndreKR commented Feb 2, 2025

telegraf-tiger bot commented Feb 2, 2025

Artifact URLs