Add iceberg catalog read support #98

Tmonster · 2025-01-29T14:59:08Z

Piggy backing off of #95

This PR adds support for attaching an Iceberg catalog and be able to read iceberg tables from a datalake. It specifically supports the following sql commands:

Create an ICEBERG secret to access your catalog
CREATE SECRET (
     TYPE ICEBERG,
     CLIENT_ID <your-catalog-client-id>,
     CLIENT_SECRET <your-catalog-client-secret>,
     ENDPOINT 'https://<your-catalog-host>/api/catalog',
     AWS_REGION 'us-east-1'
);

Attach your catalog

ATTACH 'my_catalog' AS my_datalake (TYPE ICEBERG);
-- Select your iceberg tables in the datalake via the catalog
SHOW ALL TABLES;
SELECT * FROM my_datalake.schema.table;

Some tests as well to make sure everything works. There is also a double check on the config since it's possible some catalogs won't return configs. In this case the user is required to have a second key.

samansmink

Looks great! I've added a few comments

samansmink · 2025-01-29T15:03:15Z

CMakeLists.txt

@@ -4,7 +4,7 @@ cmake_minimum_required(VERSION 2.8.12)
 set(TARGET_NAME iceberg)
 project(${TARGET_NAME})

-set(CMAKE_CXX_STANDARD 14)
+set(CMAKE_CXX_STANDARD 17)


do we really need to bump this? I would prefer not to, to avoid any CI issues

Yes I needed it for something I can't quite recall now. I think it was used for parts for "writing"

samansmink · 2025-01-29T15:05:16Z

src/catalog_api.cpp

+	}
+  return result;
+
+// throw std::runtime_error("No AWS credentials found for table");


nit: can be removed

samansmink · 2025-01-29T15:14:09Z

src/storage/ic_transaction.cpp

+	}
+}
+
+// ICConnection &ICTransaction::GetConnection() {


nit: let's just remove all commented out code. This has been copied from the uc_catalog extension which has a prototype-quality level codebase

These all come from uc_catalog extension. I left them as is

samansmink · 2025-01-29T15:21:17Z

test/sql/iceberg_catalog_read.test

+require httpfs
+
+statement ok
+CREATE SECRET (


I think this is fine for a first PR, but we should think about the UX a little here: right now there is only 1 iceberg secret which is then fetched when calling ATTACH '' AS my_datalake (TYPE ICEBERG);

However it's probably desirable to be able to do something like:

CREATE SECRET irc_secret_1 (TYPE ICEBERG, ENDPOINT 'http://127.0.0.1:8181', BEARER_TOKEN 'bla'); CREATE SECRET irc_secret_2 (TYPE ICEBERG, ENDPOINT 'http://some.other.thing.com', BEARER_TOKEN 'bla'); ATTACH 'irc_secret_1' AS irc1 (TYPE ICEBERG); ATTACH 'irc_secret_2' AS irc2 (TYPE ICEBERG);

or perhaps

CREATE SECRET irc_secret_1 (TYPE ICEBERG, SCOPE 'http://127.0.0.1:8181', BEARER_TOKEN 'bla'); CREATE SECRET irc_secret_2 (TYPE ICEBERG, SCOPE 'http://some.other.thing.com', BEARER_TOKEN 'bla'); ATTACH 'http://127.0.0.1:8181' AS irc1 (TYPE ICEBERG); ATTACH 'http://some.other.thing.com' AS irc2 (TYPE ICEBERG);

We can have some discussion on what is nicest, these are currently also open questions for other catalog extensions such as postgres and mysql i think

Named secrets should work too. If you don't provide a name, it uses __default_iceberg

samansmink · 2025-01-29T15:25:25Z

src/storage/ic_table_set.cpp

+	}
+
+	auto &ic_catalog = catalog.Cast<ICCatalog>();
+	// TODO: handle out-of-order columns using position property


This TODO is copy pasted from uc_catalog: we should figure out if this is also relevant to iceberg, if so it can stay otherwise we should remove it to avoid confusion

samansmink · 2025-01-29T15:31:43Z

src/storage/ic_table_entry.cpp

+    auto &get = (LogicalGet &)*op;
+	bind_data = std::move(get.bind_data);
+
+	return parquet_scan_function;


I think this doesn't work with tables with deletes and potentially return incorrect results. Can we add a test for this?

If it does indeed fail, we should take a look into how to solve this, or at least throw an error

This was quite hard to figure out, and I didn't work on insert/update/delete yet 😬

samansmink · 2025-01-29T15:32:11Z

src/storage/ic_table_entry.cpp

+	auto table_ref = iceberg_scan_function.bind_replace(context, bind_input);
+
+	// 1) Create a Binder and bind the parser-level TableRef -> BoundTableRef
+    auto binder = Binder::CreateBinder(context);


nit: formatting

samansmink · 2025-01-29T15:33:41Z

src/storage/ic_table_entry.cpp

+	throw NotImplementedException("BindUpdateConstraints");
+}
+
+struct MyIcebergFunctionData : public FunctionData {


this is dead code i think?

samansmink · 2025-01-29T16:04:40Z

CMakeLists.txt

    src/common/utils.cpp
    src/common/schema.cpp
    src/common/iceberg.cpp
    src/iceberg_functions/iceberg_snapshots.cpp
    src/iceberg_functions/iceberg_scan.cpp
-    src/iceberg_functions/iceberg_metadata.cpp)
+    src/iceberg_functions/iceberg_metadata.cpp
+    src/storage/ic_catalog.cpp


I feel like this structure could be a bit clearer

What do you think about renaming the IC to IRC (this code is meant for the Iceberg Rest Catalog after all) and moving everything:
src/storage/irc/irc_catalog.cpp?

This would allow us to later on create another catalog which can handle static iceberg tables, similar to duckdb/duckdb-delta#110 which could then live in src/storage/static/static_catalog.cpp

Do you want all of these files to be icr_*or just ic_catalog.cpp to be icr_catalog.cpp

I think irc_* (not icr_ 😄) everywhere that is related to the Iceberg Rest Catalog makes the most sense. We can also look to using C++ namespacing for the iceberg extension at some point, but I think just prefixing everything with IRC is fine for now.

sean-lynch · 2025-02-04T07:17:55Z

Very excited future user here! Chiming in on a review like this may be frowned upon. Please ignore if it's a distraction.

One minor request to make the ICEBERG secret agnostic of object storage provider. Rather than accepting AWS_REGION here, can that be left entirely to the existing httpfs secret? Taking some inspiration from @samansmink's feedback, maybe a future addition lets you specify which object storage provider secret to use, but at least leaving out AWS_REGION here avoids mixing up that configuration.

Tmonster · 2025-02-04T08:19:07Z

src/storage/irc_table_entry.cpp

+		                              table_data->data_source_format);
+	}
+
+	if (table_data->storage_location.find("file://") != 0) {


the hardcoded file:// maybe is unnecessary? I feel like we can omit the condition and just try to pull the credentials

Tmonster · 2025-02-04T08:20:39Z

src/storage/irc_table_entry.cpp

+			// Inject secret into secret manager scoped to this path
+			CreateSecretInfo info(OnCreateConflict::ERROR_ON_CONFLICT, SecretPersistType::TEMPORARY);
+			info.name = "__internal_ic_" + table_data->table_id;
+			info.type = "s3";


Also, based on the comment here, we should see if the spec returns information on the type of the secret so we don't have to hard code this kind of information

samansmink

Hey @Tmonster, went through the code again and added a bunch of nitpicks, other than that it looks great

samansmink · 2025-02-11T09:52:05Z

src/catalog_api.cpp

+
+IRCAPISchema IRCAPI::CreateSchema(const string &catalog, const string &internal, const string &schema, IRCCredentials credentials) {
+	throw NotImplementedException("IRCAPI::Create Schema not Implemented");
+	// string post_data = "{\"namespace\":[\"" + schema + "\"]}";


nit: maybe we can remove this commented block

samansmink · 2025-02-11T09:54:29Z

src/iceberg_extension.cpp

+	return make_uniq<ICTransactionManager>(db, ic_catalog);
+}
+
+class ICCatalogStorageExtension : public StorageExtension {


nit: IRCStorageExtension

samansmink · 2025-02-11T09:56:54Z

src/iceberg_extension.cpp

+		}
+	}
+
+	// if no secret is specified we default to the unnamed mysql secret, if it


nit: iceberg secret

samansmink · 2025-02-11T09:57:56Z

src/include/storage/irc_schema_set.hpp

+namespace duckdb {
+struct CreateSchemaInfo;
+
+class ICSchemaSet : public ICCatalogSet {


nit: IRCSchemaSet

samansmink · 2025-02-11T09:58:32Z

src/include/storage/irc_schema_entry.hpp

+namespace duckdb {
+class ICTransaction;
+
+class ICSchemaEntry : public SchemaCatalogEntry {


nit: IRCSchemaEntry

samansmink · 2025-02-11T09:59:30Z

src/storage/irc_schema_entry.cpp

+ICSchemaEntry::~ICSchemaEntry() {
+}
+
+ICTransaction &GetUCTransaction(CatalogTransaction transaction) {


Suggested change

ICTransaction &GetUCTransaction(CatalogTransaction transaction) {

IRCTransaction &GetIRCTransaction(CatalogTransaction transaction) {

samansmink · 2025-02-11T09:59:48Z

src/storage/irc_schema_entry.cpp

+			throw NotImplementedException("REPLACE ON CONFLICT in CreateView");
+		}
+	}
+	// auto &ic_transaction = GetUCTransaction(transaction);


nit: remove

samansmink · 2025-02-11T10:01:26Z

test/sql/local/iceberg_catalog_read.test

+2023-03-11	11	k
+2023-03-12	12	l
+
+# test deletes (see provision.py for where deletes occur)


scan on a table with deletes should throw a notimplementedexception for now

This is one of those cases where the whole table is rewritten and there are no point deletes. I'll see if I can add a case with point deletes that throws an error

… will clean them up though

samansmink · 2025-02-12T12:11:21Z

Great work on this, thanks a lot @ediril and @Tmonster!

ediril and others added 15 commits January 14, 2025 02:49

Merge is done

1710ff9

Rename files uc -> ic

f3d5d87

Started renaming classes

1be2fa3

FInished renaming

63ef6e0

Delete duplicate

e6f671a

Lazy load metadata for tables

46a9946

Extract precision and scale

26d9f1b

Get rid of is_loaded

645e83c

Finish implementing create/drop schema

3a63af6

Reorg classes for clarity

f24fc3a

Minor fixes

56e4c76

Rename classes for clarity

aee6c1d

Complete table create & drop

dc0ebb3

fixes so it can work with local testing information

f3007ae

add tests

22bcffe

samansmink reviewed Jan 29, 2025

View reviewed changes

Tmonster added 8 commits January 30, 2025 09:53

PR comments #1

dfb95f9

remove dead code

bffe013

renaming files and classes

92046e3

fix include errors

49c8fba

also check delete and insert statements

bb4cb29

fix formatting

40dd36c

also rename ic_catalog.cpp to icr_catalog.cpp

a8c7fe6

change ic to irc

43daceb

Tmonster commented Feb 4, 2025

View reviewed changes

Tmonster added 3 commits February 4, 2025 15:17

rename ICRCatalog to IRCatalog

c288e88

remove check for 'file://'.

bc57d82

remove check for is internal table

b2259f0

Tmonster added 10 commits February 5, 2025 14:42

more renaming ICR to IRC and remove UC references

1fa5bd8

modify test so it reads from correct files

07ef8dc

separate tests

f94a2f7

throw not implemented exceptions for drop and create statements

f49e4ab

fix formatting for test files

05b16b9

add switch expression to throw errors if table has deletes

fc5cb65

add a few more not implemented exceptions

010cf26

add tests to make sure creates/drops/alter statements are not supported

76d7946

revert removed code. add a few more tests

0aae3cf

fix indentation

e5192ed

samansmink reviewed Feb 11, 2025

View reviewed changes

Tmonster added 6 commits February 11, 2025 11:52

PR coments, more renaming to IRC and removing commented out code

dba5647

add final test. remove last piece of dead code

1db4da3

Merge branch 'main' into add_iceberg_catalog_read_support

9ad13a4

add duckdb to requirements.txt

0d5712a

finally have delete vectors working consistently in a rest catalog. I…

c5d7790

… will clean them up though

target v1.2 for main extension pipeline

85950a2

samansmink merged commit 5756233 into duckdb:main Feb 12, 2025
18 checks passed

samansmink mentioned this pull request Feb 12, 2025

Add Iceberg catalog support #95

Closed

Tmonster deleted the add_iceberg_catalog_read_support branch February 17, 2025 14:54

buremba mentioned this pull request Feb 22, 2025

TEMP TABLE does not exist when inserting into an Iceberg table buremba/universql#30

Open

This was referenced Feb 26, 2025

Iceberg REST Catalog Support #16

Open

Question about REST Catalog Implementation #102

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add iceberg catalog read support #98

Add iceberg catalog read support #98

Tmonster commented Jan 29, 2025

samansmink left a comment

samansmink Jan 29, 2025

ediril Feb 13, 2025

samansmink Jan 29, 2025

samansmink Jan 29, 2025

ediril Feb 13, 2025

samansmink Jan 29, 2025

ediril Feb 13, 2025

samansmink Jan 29, 2025

samansmink Jan 29, 2025

ediril Feb 13, 2025

samansmink Jan 29, 2025

samansmink Jan 29, 2025

samansmink Jan 29, 2025

Tmonster Feb 3, 2025

samansmink Feb 3, 2025

sean-lynch commented Feb 4, 2025 •

edited

Loading

Tmonster Feb 4, 2025

Tmonster Feb 4, 2025

samansmink left a comment

samansmink Feb 11, 2025

samansmink Feb 11, 2025

samansmink Feb 11, 2025

samansmink Feb 11, 2025

samansmink Feb 11, 2025

samansmink Feb 11, 2025

samansmink Feb 11, 2025

samansmink Feb 11, 2025

Tmonster Feb 11, 2025

samansmink commented Feb 12, 2025

	ICTransaction &GetUCTransaction(CatalogTransaction transaction) {
	IRCTransaction &GetIRCTransaction(CatalogTransaction transaction) {

Add iceberg catalog read support #98

Add iceberg catalog read support #98

Conversation

Tmonster commented Jan 29, 2025

samansmink left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sean-lynch commented Feb 4, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

samansmink left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

samansmink commented Feb 12, 2025

sean-lynch commented Feb 4, 2025 •

edited

Loading