[external contributor] Support config providers in validation by @swehner #753
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
sfc-gh-rcheng
requested review from
sfc-gh-japatel,
sfc-gh-tzhang,
sfc-gh-tjones and
a team
as code owners
sfc-gh-xhuang
approved these changes
Nov 30, 2023
|
CLA was signed and good to merge |
sfc-gh-rcheng
deleted the
swehner-support-config-providers-in-validation
branch
EduardHantig
pushed a commit
to streamkap-com/snowflake-kafka-connector
that referenced
this pull request
Feb 1, 2024
…hner (snowflakedb#753) Co-authored-by: Stefan Wehner <[email protected]>
sudeshwasnik
added a commit
to confluentinc/snowflake-kafka-connector
that referenced
this pull request
Feb 16, 2024
* [external contributor] Support config providers in validation by @swehner (snowflakedb#753) Co-authored-by: Stefan Wehner <[email protected]> * v2.1.2 release change in master (snowflakedb#757) git cherry-pick d4d0e87 * NO-SNOW: Expose the Ingest SDK MAX_CLIENT_LAG configuration in KC (snowflakedb#758) * Add authentication for max lag parameter and fix IT (snowflakedb#763) * SNOW-989387: Fix for java.lang.NoClassDefFoundError (snowflakedb#774) java.lang.NoClassDefFoundError: Could not initialize class net.snowflake.client.jdbc.internal.apache.arrow.memory.RootAllocator occurs when the JDBC driver tries to process the result of the offset migration query. This is a known long-standing issue in the JDBC driver so a workaround is introduced with this fix. * [SNOW-1015644] Update ingest sdk to 2.0.5 (snowflakedb#778) * SNOW-943288, SNOW-938379: Fix two schema evolution cases that could cause non-exactly once delivery (snowflakedb#775) Fix two exactly once delivery behavior issues with Snowpipe Streaming: - When there're gaps between offsets due to records being put into the DLQ or NULL records being skipped, the current logic doesn't work after a channel reopening event (e.g. schema evolution) since it expects continuous offsets in order to guarantee exactly once delivery, causing ingestion to stop. - When a flush is triggered by size or rowcount threshold, it's possible that only partial rows in the buffer are flushed and with a channel reopening event (e.g. schema evolution), currently the leftover rows in the buffer are still being considered which is causing the offset to be advanced and skip some offsets. The fix in this PR makes sure we won't flush any leftover rows in the buffer, they will all be skipped and discarded. The next batch should start from the expected offset after the offset reset as part of the channel reopening event. * resolves issue with generating and building java libary (snowflakedb#780) * [SNOW-88848] Update readme with Snowflake documentation (snowflakedb#781) * Update version to 2.2.0 for release (snowflakedb#779) * Downgrade JDBC to 3.13.30 (snowflakedb#783) * SNOW-922795 Added third-party-licenses to .zip distribution (snowflakedb#784) * PROD-39429 Add PR template (snowflakedb#786) --------- Co-authored-by: revi cheng <[email protected]> Co-authored-by: Stefan Wehner <[email protected]> Co-authored-by: Toby Zhang <[email protected]> Co-authored-by: Waleed Fateem <[email protected]> Co-authored-by: Greg Jachimko <[email protected]> Co-authored-by: Michał Bobowski <[email protected]> Co-authored-by: Jay Patel <[email protected]>
sudeshwasnik
pushed a commit
to confluentinc/snowflake-kafka-connector
that referenced
this pull request
Feb 16, 2024
…hner (snowflakedb#753) Co-authored-by: Stefan Wehner <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Copy @swehner 's PR for github actions runs.
@sfc-gh-xhuang please confirm that CLA was signed
Original PR description:
I'm trying to use different config providers in my snowflake connector, but get an error during validation because the snowflake private key is not valid.
This is because validation seems to receive the raw config values (not running through configProviders which seems iffy).
In any case there was already a workaround to support ${file: as a config provider value and to skip validation. I've extended it to include any configProvider with a different name.
Fixes issue #748