Use USE_VECTORIZED_SCANNER=True for loading parquet files to Snowflake

[VioletM]

Feature description

Snowflake has a way of optimizing the loading of parquet data with USE_VECTORIZED_SCANNER option.

https://www.snowflake.com/en/engineering-blog/loading-terabytes-into-snowflake-speeds-feeds-techniques/
https://docs.snowflake.com/en/sql-reference/sql/copy-into-table#label-use-vectorized-scanner

The feature request is to enable this by default (it's already marked as TODO:

dlt/dlt/destinations/impl/snowflake/snowflake.py

Line 176 in ae18acb

# TODO: USE_VECTORIZED_SCANNER inserts null strings into VARIANT JSON

)

Important note:

You can only enable the vectorized scanner if the following conditions are met:

The ON_ERROR option must be set to ABORT_STATEMENT or SKIP_FILE.

The other values, CONTINUE, SKIP_FILE_num, 'SKIP_FILE_num%' are not supported.

Are you a dlt user?

Yes, I run dlt in production.

Implementation details

1. • add an option to do that to SnowflakeConfiguration, disabled by default
2. • investigate if test_parquet_loading test works with option enabled,
3. • write a specialized test for snowflake based on test above
4. • document the new option