Apache X table with Snowflake and glue

[soumilshah1995]

I am able to sync items from snowflake and build metadata for other table formats

sourceFormat: ICEBERG

targetFormats:
  - DELTA
  - HUDI

datasets:
  -
    tableBasePath: s3://soumilshah-dev-1995/warehouse/snowflake_tables/customers/
    tableDataPath: s3://soumilshah-dev-1995/warehouse/snowflake_tables/customers/data
    tableName: customers
    namespace: tempdb.public

CMD ["java", "-cp", "iceberg-spark-runtime-3.3_2.12-1.4.2.jar:xtable-utilities-0.1.0-SNAPSHOT-bundled.jar:snowflake-jdbc-3.13.28.jar:iceberg-aws-1.4.2.jar:bundle-2.23.9.jar", "org.apache.xtable.utilities.RunSync", "--datasetConfig", "config.yaml", "--icebergCatalogConfig", "snowflake_catalog.yaml"]

Now I want to sync with glue catalog
glue_config.yaml

sourceFormat: ICEBERG

targetFormats:
  - DELTA
  - HUDI

datasets:
  -
    tableBasePath: s3://soumilshah-dev-1995/warehouse/snowflake_tables/customers/
    tableName: customers
    namespace: icebergdb

glue_catalog_sync.yaml

catalogImpl: org.apache.iceberg.aws.glue.GlueCatalog
catalogName: onetable
catalogOptions:
  io-impl: org.apache.iceberg.aws.s3.S3FileIO
  warehouse: s3://soumilshah-dev-1995/warehouse

CMD ["java", "-cp", "iceberg-spark-runtime-3.3_2.12-1.4.2.jar:xtable-utilities-0.1.0-SNAPSHOT-bundled.jar:snowflake-jdbc-3.13.28.jar:iceberg-aws-1.4.2.jar:bundle-2.23.9.jar", "org.apache.xtable.utilities.RunSync", "--datasetConfig", "glue_config.yaml", "--icebergCatalogConfig", "glue_catalog_sync.yaml"]

024-10-28 21:10:33 INFO  org.apache.xtable.utilities.RunSync:148 - Running sync for basePath s3://soumilshah-dev-1995/warehouse/snowflake_tables/customers/ for following table formats [DELTA, HUDI]
2024-10-28 21:10:33 WARN  org.apache.hadoop.util.NativeCodeLoader:60 - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2024-10-28 21:10:34 WARN  org.apache.hadoop.metrics2.impl.MetricsConfig:136 - Cannot locate configuration: tried hadoop-metrics2-s3a-file-system.properties,hadoop-metrics2.properties
2024-10-28 21:10:34 WARN  org.apache.hadoop.fs.s3a.SDKV2Upgrade:39 - Directly referencing AWS SDK V1 credential provider com.amazonaws.auth.DefaultAWSCredentialsProviderChain. AWS SDK V1 credential providers will be removed once S3A is upgraded to SDK V2
2024-10-28 21:10:34 INFO  org.apache.spark.sql.delta.storage.DelegatingLogStore:60 - LogStore `LogStoreAdapter(io.delta.storage.S3SingleDriverLogStore)` is used for scheme `s3`
2024-10-28 21:10:35 INFO  org.apache.spark.sql.delta.DeltaLog:60 - Creating initial snapshot without metadata, because the directory is empty
2024-10-28 21:10:35 INFO  org.apache.spark.sql.delta.InitialSnapshot:60 - [tableId=c3b344db-a873-45bb-b55e-1ed3b1282c7c] Created snapshot InitialSnapshot(path=s3://soumilshah-dev-1995/warehouse/snowflake_tables/customers/_delta_log, version=-1, metadata=Metadata(ae1fbf45-a01f-4bb0-b51d-e057fa2ff83e,null,null,Format(parquet,Map()),null,List(),Map(),Some(1730149835750)), logSegment=LogSegment(s3://soumilshah-dev-1995/warehouse/snowflake_tables/customers/_delta_log,-1,List(),None,-1), checksumOpt=None)
2024-10-28 21:10:36 INFO  org.apache.hudi.common.table.HoodieTableMetaClient:133 - Loading HoodieTableMetaClient from s3://soumilshah-dev-1995/warehouse/snowflake_tables/customers/
2024-10-28 21:10:36 INFO  org.apache.xtable.hudi.HudiTableManager:73 - Hudi table does not exist, will be created on first sync
2024-10-28 21:10:36 INFO  org.apache.xtable.conversion.ConversionController:240 - No previous InternalTable sync for target. Falling back to snapshot sync.
2024-10-28 21:10:36 INFO  org.apache.xtable.conversion.ConversionController:240 - No previous InternalTable sync for target. Falling back to snapshot sync.
2024-10-28 21:10:36 INFO  org.apache.iceberg.CatalogUtil:302 - Loading custom FileIO implementation: org.apache.iceberg.aws.s3.S3FileIO
2024-10-28 21:10:37 ERROR org.apache.xtable.utilities.RunSync:171 - Error running sync for s3://soumilshah-dev-1995/warehouse/snowflake_tables/customers/
org.apache.iceberg.exceptions.NoSuchTableException: Table does not exist: icebergdb.customers
	at org.apache.iceberg.BaseMetastoreCatalog.loadTable(BaseMetastoreCatalog.java:53) ~[iceberg-spark-runtime-3.3_2.12-1.4.2.jar:?]
	at org.apache.xtable.iceberg.IcebergTableManager.lambda$getTable$0(IcebergTableManager.java:57) ~[xtable-utilities-0.1.0-SNAPSHOT-bundled.jar:0.1.0-SNAPSHOT]
	at java.util.Optional.map(Optional.java:215) ~[?:1.8.0_432]
	at org.apache.xtable.iceberg.IcebergTableManager.getTable(IcebergTableManager.java:57) ~[xtable-utilities-0.1.0-SNAPSHOT-bundled.jar:0.1.0-SNAPSHOT]
	at org.apache.xtable.iceberg.IcebergConversionSource.initSourceTable(IcebergConversionSource.java:81) ~[xtable-utilities-0.1.0-SNAPSHOT-bundled.jar:0.1.0-SNAPSHOT]
	at org.apache.xtable.iceberg.IcebergConversionSource.getSourceTable(IcebergConversionSource.java:60) ~[xtable-utilities-0.1.0-SNAPSHOT-bundled.jar:0.1.0-SNAPSHOT]
	at org.apache.xtable.iceberg.IcebergConversionSource.getCurrentSnapshot(IcebergConversionSource.java:121) ~[xtable-utilities-0.1.0-SNAPSHOT-bundled.jar:0.1.0-SNAPSHOT]
	at org.apache.xtable.spi.extractor.ExtractFromSource.extractSnapshot(ExtractFromSource.java:38) ~[xtable-utilities-0.1.0-SNAPSHOT-bundled.jar:0.1.0-SNAPSHOT]
	at org.apache.xtable.conversion.ConversionController.syncSnapshot(ConversionController.java:183) ~[xtable-utilities-0.1.0-SNAPSHOT-bundled.jar:0.1.0-SNAPSHOT]
	at org.apache.xtable.conversion.ConversionController.sync(ConversionController.java:121) ~[xtable-utilities-0.1.0-SNAPSHOT-bundled.jar:0.1.0-SNAPSHOT]
	at org.apache.xtable.utilities.RunSync.main(RunSync.java:169) [xtable-utilities-0.1.0-SNAPSHOT-bundled.jar:0.1.0-SNAPSHOT]
sshah@D345YG773G sync % 

question I dont want to use glue crawler can I do so using above ?

[the-other-tim-brown]

@soumilshah1995 since Iceberg is the source, it is required to already be in the Glue Catalog for Iceberg. Can you validate that the naming matches?

[soumilshah1995]

Hi there,
Let me explain my use case for clarity.
I've set up an Iceberg table in Snowflake as follows:

CREATE OR REPLACE ICEBERG TABLE tempdb.public.customers (
    id VARCHAR,
    name VARCHAR,
    age VARCHAR,
    city VARCHAR,
    create_ts VARCHAR
)
CATALOG='SNOWFLAKE'
EXTERNAL_VOLUME='ext_vol'
BASE_LOCATION='snowflake_tables/customers';

INSERT INTO tempdb.public.customers (id, name, age, city, create_ts)
VALUES
    (1, 'John', 25, 'NYC', '2023-09-28 00:00:00'),
    (2, 'Emily', 30, 'SFO', '2023-09-28 00:00:00'),
    (3, 'Michael', 35, 'ORD', '2023-09-28 00:00:00'),
    (4, 'Andrew', 40, 'NYC', '2023-10-28 00:00:00'),
    (5, 'Bob', 28, 'SEA', '2023-09-23 00:00:00'),
    (6, 'Charlie', 31, 'DFW', '2023-08-29 00:00:00');

SELECT * FROM tempdb.public.customers

After creating the table, I ran a sync command to build the metadata in Snowflake using the following command:

CMD ["java", "-cp", "iceberg-spark-runtime-3.3_2.12-1.4.2.jar:xtable-utilities-0.1.0-SNAPSHOT-bundled.jar:snowflake-jdbc-3.13.28.jar:iceberg-aws-1.4.2.jar:bundle-2.23.9.jar", "org.apache.xtable.utilities.RunSync", "--datasetConfig", "config.yaml", "--icebergCatalogConfig", "snowflake_catalog.yaml"]

Now, I want to perform a second sync to register this metadata in AWS Glue without needing to run the Glue crawlers. The table was created in Snowflake as a managed Iceberg table, and I initially used snowflake_catalog.yaml and config.yaml to set up the metadata.
For the Glue sync, I’m planning to use glue_catalog_sync.yaml and glue_config.yaml files as follows:

CMD ["java", "-cp", "iceberg-spark-runtime-3.3_2.12-1.4.2.jar:xtable-utilities-0.1.0-SNAPSHOT-bundled.jar:snowflake-jdbc-3.13.28.jar:iceberg-aws-1.4.2.jar:bundle-2.23.9.jar", "org.apache.xtable.utilities.RunSync", "--datasetConfig", "glue_config.yaml", "--icebergCatalogConfig", "glue_catalog_sync.yaml"]

My question: Is it possible to run the second sync to register the metadata in Glue directly using these files without triggering Glue crawlers?

OR

catalog.template.yaml

catalogImpl: org.apache.iceberg.snowflake.SnowflakeCatalog
catalogName: onetable
catalogOptions:
  io-impl: org.apache.iceberg.aws.s3.S3FileIO
  warehouse: ${S3_BUCKET}/warehouse
  uri: jdbc:snowflake://${SNOWFLAKE_ACCOUNT}.snowflakecomputing.com
  jdbc.user: ${SNOWFLAKE_USER}
  jdbc.password: ${SNOWFLAKE_PASSWORD}

glue_catalog_sync.yaml

catalogImpl: org.apache.iceberg.aws.glue.GlueCatalog
catalogName: onetable
catalogOptions:
  io-impl: org.apache.iceberg.aws.s3.S3FileIO
  warehouse: s3://soumilshah-dev-1995/warehouse

Can I run Both together ?

is there a way to sync to multiple catalog ?

Thank you!

[the-other-tim-brown]

In the original comment, I see that Iceberg is the source for the second sync to Glue. You can try to convert that a Delta or Hudi source and an Iceberg target with the catalog.yaml for Glue. We have never tried this case before though so not sure if it will work. Typically, you only have one catalog for Iceberg so be careful and make sure only one is used on the writer path to resolve any writer conflicts.