From 9462164a9154ae4cc7a2105ad80e0af977dd9365 Mon Sep 17 00:00:00 2001 From: Zach Schuermann Date: Mon, 24 Feb 2025 15:01:34 -0800 Subject: [PATCH 1/4] release 0.7.0 --- CHANGELOG.md | 89 +++++++++++++++++++++++++++++++++++++++++++++++ Cargo.toml | 2 +- README.md | 4 +-- ffi/Cargo.toml | 2 +- kernel/Cargo.toml | 2 +- 5 files changed, 94 insertions(+), 5 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 4d403a1fd..29ef0e300 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,5 +1,94 @@ # Changelog +## [v0.7.0](https://github.com/delta-io/delta-kernel-rs/tree/v0.7.0/) (2025-02-24) + +[Full Changelog](https://github.com/delta-io/delta-kernel-rs/compare/v0.6.1...v0.7.0) + + +### ๐Ÿš€ Features / new APIs + +1. Read `partition_values` in `RemoveVisitor` and remove `break` in `RowVisitor` for `RemoveVisitor` ([#633]) +2. Add the in-commit timestamp field to CommitInfo ([#581]) +3. Support NOT and column expressions in eval_sql_where ([#653]) +4. Feat!(ffi): support field nullability in schema visitor ([#656]) +5. Add check for schema read compatibility ([#554]) +6. Introduce `TableConfiguration` to jointly manage metadata, protocol, and table properties ([#644]) +7. Feat!(ffi): expose metadata in SchemaEngineVisitor ffi api ([#659]) +8. Add visitor `SidecarVisitor` and `Sidecar` action struct ([#673]) +9. Add in-commit timestamps table properties ([#558]) +10. Feat!(ffi): new visit_schema FFI and rename old visit_schema to visit_snapshot_schema ([#683]) +11. Introduce feature flags to select major arrow versions ([#654]) +12. Support writing to not only 3/7 protocol ([#693]) +13. Feat!(ffi): remove `visit_snapshot_schema`, add `logical_schema` ([#709]) + +### ๐Ÿ› Bug Fixes + +1. Release script publishing fixes ([#638]) +2. Incomplete multi-part checkpoint handling when no hint is provided ([#641]) +3. Consistent PartialEq for Scalar ([#677]) +4. Cargo fmt does not handle mods defined in macros ([#676]) +5. Ensure properly nested null masks for parquet reads ([#692]) +6. Handle predicates on non-nullable columns without stats ([#700]) + +### ๐Ÿ“š Documentation + +1. Update readme to reflect tracing feature is needed for read-table ([#619]) +2. Clarify `JsonHandler` semantics on EngineData ordering ([#635]) + +### ๐Ÿšœ Refactor + +1. Make [non] nullable struct fields easier to create ([#646]) + +### ๐Ÿงช Testing + +1. Port cdf tests from delta-spark to kernel ([#611]) + +### โš™๏ธ Chores/CI + +1. Fix some typos ([#643]) + +### Other + +1. Make eval_sql_where available to DefaultPredicateEvaluator ([#627]) +2. Part 1, Read transforms via expressions: Just compute the expression and return it. ([#607]) +3. Part 2: propagate transform in visit_scan_files ([#612]) +4. Part 3 of expression based transform: Use computed transform ([#613]) +5. Part 4: read_table.c uses transform in ffi ([#614]) +6. Support --all-features again ([#708]) + + +[#638]: https://github.com/delta-io/delta-kernel-rs/pull/638 +[#643]: https://github.com/delta-io/delta-kernel-rs/pull/643 +[#619]: https://github.com/delta-io/delta-kernel-rs/pull/619 +[#635]: https://github.com/delta-io/delta-kernel-rs/pull/635 +[#633]: https://github.com/delta-io/delta-kernel-rs/pull/633 +[#611]: https://github.com/delta-io/delta-kernel-rs/pull/611 +[#581]: https://github.com/delta-io/delta-kernel-rs/pull/581 +[#646]: https://github.com/delta-io/delta-kernel-rs/pull/646 +[#627]: https://github.com/delta-io/delta-kernel-rs/pull/627 +[#641]: https://github.com/delta-io/delta-kernel-rs/pull/641 +[#653]: https://github.com/delta-io/delta-kernel-rs/pull/653 +[#607]: https://github.com/delta-io/delta-kernel-rs/pull/607 +[#656]: https://github.com/delta-io/delta-kernel-rs/pull/656 +[#554]: https://github.com/delta-io/delta-kernel-rs/pull/554 +[#644]: https://github.com/delta-io/delta-kernel-rs/pull/644 +[#659]: https://github.com/delta-io/delta-kernel-rs/pull/659 +[#612]: https://github.com/delta-io/delta-kernel-rs/pull/612 +[#677]: https://github.com/delta-io/delta-kernel-rs/pull/677 +[#676]: https://github.com/delta-io/delta-kernel-rs/pull/676 +[#673]: https://github.com/delta-io/delta-kernel-rs/pull/673 +[#613]: https://github.com/delta-io/delta-kernel-rs/pull/613 +[#558]: https://github.com/delta-io/delta-kernel-rs/pull/558 +[#692]: https://github.com/delta-io/delta-kernel-rs/pull/692 +[#700]: https://github.com/delta-io/delta-kernel-rs/pull/700 +[#683]: https://github.com/delta-io/delta-kernel-rs/pull/683 +[#654]: https://github.com/delta-io/delta-kernel-rs/pull/654 +[#693]: https://github.com/delta-io/delta-kernel-rs/pull/693 +[#614]: https://github.com/delta-io/delta-kernel-rs/pull/614 +[#709]: https://github.com/delta-io/delta-kernel-rs/pull/709 +[#708]: https://github.com/delta-io/delta-kernel-rs/pull/708 + + ## [v0.6.1](https://github.com/delta-io/delta-kernel-rs/tree/v0.6.1/) (2025-01-10) [Full Changelog](https://github.com/delta-io/delta-kernel-rs/compare/v0.6.0...v0.6.1) diff --git a/Cargo.toml b/Cargo.toml index aec38fc78..b85704310 100644 --- a/Cargo.toml +++ b/Cargo.toml @@ -20,7 +20,7 @@ license = "Apache-2.0" repository = "https://github.com/delta-io/delta-kernel-rs" readme = "README.md" rust-version = "1.80" -version = "0.6.1" +version = "0.7.0" [workspace.dependencies] object_store = { version = ">=0.11, <0.12" } diff --git a/README.md b/README.md index 23eff8770..1cddea1a8 100644 --- a/README.md +++ b/README.md @@ -43,10 +43,10 @@ consumer's own `Engine` trait, the kernel has a feature flag to enable a default ```toml # fewer dependencies, requires consumer to implement Engine trait. # allows consumers to implement their own in-memory format -delta_kernel = "0.6.1" +delta_kernel = "0.7.0" # or turn on the default engine, based on arrow -delta_kernel = { version = "0.6.1", features = ["default-engine"] } +delta_kernel = { version = "0.7.0", features = ["default-engine"] } ``` ### Feature flags diff --git a/ffi/Cargo.toml b/ffi/Cargo.toml index d588427b0..fe0aa97eb 100644 --- a/ffi/Cargo.toml +++ b/ffi/Cargo.toml @@ -25,7 +25,7 @@ delta_kernel = { path = "../kernel", default-features = false, features = [ "arrow", "developer-visibility", ] } -delta_kernel_ffi_macros = { path = "../ffi-proc-macros", version = "0.6.1" } +delta_kernel_ffi_macros = { path = "../ffi-proc-macros", version = "0.7.0" } [build-dependencies] cbindgen = "0.28" diff --git a/kernel/Cargo.toml b/kernel/Cargo.toml index 8904ccaa5..886ba3c60 100644 --- a/kernel/Cargo.toml +++ b/kernel/Cargo.toml @@ -51,7 +51,7 @@ uuid = "1.10.0" z85 = "3.0.5" # bring in our derive macros -delta_kernel_derive = { path = "../derive-macros", version = "0.6.1" } +delta_kernel_derive = { path = "../derive-macros", version = "0.7.0" } # used for developer-visibility visibility = "0.1.1" From a88724a235bdbc3e8080d81d41eb081b1082c30b Mon Sep 17 00:00:00 2001 From: Zach Schuermann Date: Mon, 24 Feb 2025 15:42:04 -0800 Subject: [PATCH 2/4] fix changelog --- CHANGELOG.md | 52 ++++++++++++++++++++++++++-------------------------- 1 file changed, 26 insertions(+), 26 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 29ef0e300..6d39e50c9 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,31 +4,39 @@ [Full Changelog](https://github.com/delta-io/delta-kernel-rs/compare/v0.6.1...v0.7.0) +### ๐Ÿ—๏ธ Breaking changes +1. Read transforms are now communicated via expressions ([#607], [#612], [#613], [#614]) This includes: + - `ScanData` now includes a third tuple field: a row-indexed vector of transforms to apply to the `EngineData`. + - Adds a new `scan::state::transform_to_logical` function that encapsulates the boilerplate of applying the transform expression + - Removes `scan_action_iter` API and `logical_to_physical` API + - Removes `column_mapping_mode` from `GlobalScanState` + - ffi: Removes `add_partition_columns` in arrow.c + - ffi: exposes methods to get an expression evaluator and evaluate an expression from c + - ffi: adds an `apply_transform` function in arrow.c +2. ffi: support field nullability in schema visitor ([#656]) +3. ffi: expose metadata in SchemaEngineVisitor ffi api ([#659]) +4. ffi: new `visit_schema` FFI now operates on a `Schema` instead of a `Snapshot` ([#683], [#709]) +5. Introduced feature flags (`arrow_54` and `arrow_53`) to select major arrow versions ([#654], [#708]) ### ๐Ÿš€ Features / new APIs 1. Read `partition_values` in `RemoveVisitor` and remove `break` in `RowVisitor` for `RemoveVisitor` ([#633]) 2. Add the in-commit timestamp field to CommitInfo ([#581]) 3. Support NOT and column expressions in eval_sql_where ([#653]) -4. Feat!(ffi): support field nullability in schema visitor ([#656]) -5. Add check for schema read compatibility ([#554]) -6. Introduce `TableConfiguration` to jointly manage metadata, protocol, and table properties ([#644]) -7. Feat!(ffi): expose metadata in SchemaEngineVisitor ffi api ([#659]) -8. Add visitor `SidecarVisitor` and `Sidecar` action struct ([#673]) -9. Add in-commit timestamps table properties ([#558]) -10. Feat!(ffi): new visit_schema FFI and rename old visit_schema to visit_snapshot_schema ([#683]) -11. Introduce feature flags to select major arrow versions ([#654]) -12. Support writing to not only 3/7 protocol ([#693]) -13. Feat!(ffi): remove `visit_snapshot_schema`, add `logical_schema` ([#709]) +4. Add check for schema read compatibility ([#554]) +5. Introduce `TableConfiguration` to jointly manage metadata, protocol, and table properties ([#644]) +6. Add visitor `SidecarVisitor` and `Sidecar` action struct ([#673]) +7. Add in-commit timestamps table properties ([#558]) +8. Support writing to writer version 1 ([#693]) +9. ffi: new `logical_schema` FFI to get the logical schema of a snapshot ([#709]) ### ๐Ÿ› Bug Fixes -1. Release script publishing fixes ([#638]) -2. Incomplete multi-part checkpoint handling when no hint is provided ([#641]) -3. Consistent PartialEq for Scalar ([#677]) -4. Cargo fmt does not handle mods defined in macros ([#676]) -5. Ensure properly nested null masks for parquet reads ([#692]) -6. Handle predicates on non-nullable columns without stats ([#700]) +1. Incomplete multi-part checkpoint handling when no hint is provided ([#641]) +2. Consistent PartialEq for Scalar ([#677]) +3. Cargo fmt does not handle mods defined in macros ([#676]) +4. Ensure properly nested null masks for parquet reads ([#692]) +5. Handle predicates on non-nullable columns without stats ([#700]) ### ๐Ÿ“š Documentation @@ -38,6 +46,7 @@ ### ๐Ÿšœ Refactor 1. Make [non] nullable struct fields easier to create ([#646]) +2. Make eval_sql_where available to DefaultPredicateEvaluator ([#627]) ### ๐Ÿงช Testing @@ -46,16 +55,7 @@ ### โš™๏ธ Chores/CI 1. Fix some typos ([#643]) - -### Other - -1. Make eval_sql_where available to DefaultPredicateEvaluator ([#627]) -2. Part 1, Read transforms via expressions: Just compute the expression and return it. ([#607]) -3. Part 2: propagate transform in visit_scan_files ([#612]) -4. Part 3 of expression based transform: Use computed transform ([#613]) -5. Part 4: read_table.c uses transform in ffi ([#614]) -6. Support --all-features again ([#708]) - +2. Release script publishing fixes ([#638]) [#638]: https://github.com/delta-io/delta-kernel-rs/pull/638 [#643]: https://github.com/delta-io/delta-kernel-rs/pull/643 From c981c702b5e133172b83c1e39bdfaa8550dd7692 Mon Sep 17 00:00:00 2001 From: Zach Schuermann Date: Mon, 24 Feb 2025 15:54:59 -0800 Subject: [PATCH 3/4] fix changelog --- CHANGELOG.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 6d39e50c9..4e4cfd7b2 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -10,9 +10,9 @@ - Adds a new `scan::state::transform_to_logical` function that encapsulates the boilerplate of applying the transform expression - Removes `scan_action_iter` API and `logical_to_physical` API - Removes `column_mapping_mode` from `GlobalScanState` - - ffi: Removes `add_partition_columns` in arrow.c - ffi: exposes methods to get an expression evaluator and evaluate an expression from c - - ffi: adds an `apply_transform` function in arrow.c + - read-table example: Removes `add_partition_columns` in arrow.c + - read-table example: adds an `apply_transform` function in arrow.c 2. ffi: support field nullability in schema visitor ([#656]) 3. ffi: expose metadata in SchemaEngineVisitor ffi api ([#659]) 4. ffi: new `visit_schema` FFI now operates on a `Schema` instead of a `Snapshot` ([#683], [#709]) From 896da63cb95a43f5ba8561d7cd635bf7f7d5eca7 Mon Sep 17 00:00:00 2001 From: Zach Schuermann Date: Mon, 24 Feb 2025 18:18:17 -0800 Subject: [PATCH 4/4] add changelog --- CHANGELOG.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 4e4cfd7b2..49f89018e 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -16,7 +16,7 @@ 2. ffi: support field nullability in schema visitor ([#656]) 3. ffi: expose metadata in SchemaEngineVisitor ffi api ([#659]) 4. ffi: new `visit_schema` FFI now operates on a `Schema` instead of a `Snapshot` ([#683], [#709]) -5. Introduced feature flags (`arrow_54` and `arrow_53`) to select major arrow versions ([#654], [#708]) +5. Introduced feature flags (`arrow_54` and `arrow_53`) to select major arrow versions ([#654], [#708], [#717]) ### ๐Ÿš€ Features / new APIs @@ -87,6 +87,7 @@ [#614]: https://github.com/delta-io/delta-kernel-rs/pull/614 [#709]: https://github.com/delta-io/delta-kernel-rs/pull/709 [#708]: https://github.com/delta-io/delta-kernel-rs/pull/708 +[#717]: https://github.com/delta-io/delta-kernel-rs/pull/717 ## [v0.6.1](https://github.com/delta-io/delta-kernel-rs/tree/v0.6.1/) (2025-01-10)