Kernel: Add a conversion utility to convert Iceberg schema to Delta schema #4075
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Comment on lines
+392
to
+398
| This product includes code from Apache Iceberg. | ||
|
|
||
| * io.delta.kernel.internal.util.IcebergTypeToStructType visitor implementation based on org.apache.iceberg.spark.TypeToSparkType | ||
|
|
||
| Copyright: 2018-2024 The Apache Software Foundation | ||
| Home page: https://iceberg.apache.org/ | ||
| License: https://www.apache.org/licenses/LICENSE-2.0 |
There was a problem hiding this comment.
Need to make sure this attribution is done correctly
Comment on lines
+23
to
+24
| Apache Iceberg | ||
| Copyright 2017-2024 The Apache Software Foundation |
There was a problem hiding this comment.
Same as above, I think this change to the notice is correct, currently the kernel module doesn't have its own License/Notice so I added this to the root level.
amogh-jahagirdar
force-pushed
the
iceberg-to-delta-schema-util
branch
from
d04e988 to
ea5d869
Compare
Comment on lines
+580
to
+581
| "org.apache.iceberg" % "iceberg-api" % "1.6.1", | ||
| "org.apache.iceberg" % "iceberg-core" % "1.6.1", |
There was a problem hiding this comment.
Can't use 1.7.1 at the moment because Delta only builds with JDK 8 at the moment as far as I can tell, with the exception of Spark master which is using JDk 17 (spark 4.0 requires JDK 11+)
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This change adds a
icebergToDeltaSchemautil function to SchemaUtils which converts an Iceberg Schema to a Delta schema.Which Delta project/connector is this regarding?
Description
This change adds a
icebergToDeltaSchemautil function to SchemaUtils which converts an Iceberg Schema to a Delta schema. The utility function should be a part of kernel to enable any consumers of kernel to be able to do this kind of metadata conversion.How was this patch tested?
Added unit tests.
Does this PR introduce any user-facing changes?
New utility API added