Iceberg/sink: Iceberg Sink with Snapshot for Append-Only data

[laskoviymishka]

Implement Iceberg Sink with Snapshot and Append-Only Support

Feature Request

Develop an Iceberg sink, focusing on writing data into Iceberg tables with snapshot isolation and append-only mode. This implementation will not include replication or upserts in its initial scope.
Inspired by Databricks Iceberg Kafka Connect and kafka-delta-ingest for architecture and implementation insights.

---

Scope and Goals

✅ Support appending new data files to Iceberg tables.
✅ Ensure snapshot isolation when writing new data.
✅ Use the Iceberg commit protocol to guarantee atomic writes.
✅ Implement Parquet file writing as the primary format.
✅ Incorporate basic data transformations during ingestion.
✅ Implement metrics reporting for monitoring purposes.

🚫 Replication and CDC handling are out of scope.
🚫 No upsert or merge-on-read logic in this phase.

---

Implementation Details

• Commit Model:

  • Each write operation should create a new snapshot in Iceber
  • Transactions should ensure atomic addition of new data file

• Data Format & Partitioning:

  • Support Parquet format for writing data
  • Implement basic partitioning support based on Iceberg schem

• Data Transformation:

  • Incorporate simple transformations during ingestion, such as deriving partition columns or adding metadat
  • Consider using transformation

• Configuration Options:

  • Target Iceberg table
  • File format (initially Parquet
  • Partitioning strateg
  • Transformation rule

• Error Handling & Retries:

  • Ensure robust failure recovery mechanism
  • Implement logging and monitoring for write
  • Consider implementing a dead letter queue for problematic message

• Metrics Reporting:

  • Integrate metrics reporting to monitor ingestion performance

---

References & Inspiration

• Databricks Iceberg Kafka Connect
• kafka-delta-ingest
• Apache Iceberg Table Spec
• Iceberg Commit & Snapshot Mechanism
• Iceberg kafak sink design doc

[laskoviymishka]

Blocked apache/iceberg-go#287