chore: upgrade to DataFusion 46.0.0 (WORK IN PROGRESS) #3261

alamb · 2025-02-24T19:29:55Z

Description

chore: upgrade to DataFusion 46.0.0

This is a work in progress. I am testing the effects of upgrading to DataFusion 46

Related Issue(s)

related to Release DataFusion 46.0.0 apache/datafusion#14123

Documentation

github-actions · 2025-02-24T19:30:16Z

ACTION NEEDED

delta-rs follows the Conventional Commits specification for release automation.

The PR title and description are used as the merge commit message. Please update your PR title and description to match the specification.

ion-elgreco · 2025-02-24T19:36:39Z

crates/sql/src/planner.rs

@@ -44,6 +44,7 @@ impl<'a, S: ContextProvider> DeltaSqlToRel<'a, S> {
                        enable_ident_normalization: self.options.enable_ident_normalization,
                        support_varchar_with_length: false,
                        enable_options_value_normalization: false,
+                        collect_spans: false,


What is spans?

The source code locations for expressions:

https://docs.rs/sqlparser/latest/sqlparser/tokenizer/struct.Span.html

We are starting to gather / plumb this into DataFusion. This particular setting means the sql planner won't try and pass the span information along. The only thing the spans are used for now is debug / help messages

Ah ok thanks for explaining :) I learned something new about datafusion!

FWIW this is a new feature that was starting to be added in DataFusion 46

alamb · 2025-02-24T19:44:50Z

Looks like from CI we also need to update rust to be 1.82

https://github.com/delta-io/delta-rs/actions/runs/13506223447/job/37736308124?pr=3261

error: rustc 1.81.0 is not supported by the following packages:
[email protected] requires rustc 1.82.0
[email protected] requires rustc 1.82.0
...

alamb · 2025-02-24T21:57:26Z

I have a few cleanup PRs I would plan to make as I work on this. Here is the first one:

chore: fix some compiler warnings #3262

alamb · 2025-02-25T17:58:43Z

I also started breaking this PR up into some smaller ones:

alamb · 2025-02-26T12:29:03Z

crates/core/src/delta_datafusion/mod.rs

    #[derive(Default)]
-    struct ParquetPredicateVisitor {
+    struct ParquetVisitor {


I had to change the way the parquet scan information was found in the two visitors, and I combined them together at the same time as they were mostly boiler plate copy/paste

alamb · 2025-02-26T20:40:41Z

I filed a ticket in datafusion explaining the current test failures

Error projecting statistics in DataSourceExec apache/datafusion#14905

Signed-off-by: Andrew Lamb <[email protected]>

alamb · 2025-02-28T15:58:56Z

crates/sql/src/planner.rs

-                        enable_options_value_normalization: false,
-                    },
-                );
+                let parser_options = ParserOptions::new()


This uses the nice new API from @kosiew in

Implement builder style API for ParserOptions apache/datafusion#14887

Signed-off-by: Andrew Lamb <[email protected]>

alamb · 2025-02-28T16:16:57Z

.github/workflows/build.yml

@@ -20,7 +20,7 @@ jobs:
        uses: actions-rs/toolchain@v1
        with:
          profile: default
-          toolchain: '1.81'
+          toolchain: '1.82'


DataFusion 46 requires Rust 1.82

You can see CI fail without these changes
https://github.com/delta-io/delta-rs/actions/runs/13591787541/job/38000195037?pr=3261

I don't know what the MSRV policy in delta is so we probably can't merge this PR until it is ok to increase MSRV in delta

alamb · 2025-02-28T18:42:07Z

I think this PR will pass -- maybe someone could trigger the CI so I can show a clean run?

codecov · 2025-02-28T18:53:49Z

Codecov Report

Attention: Patch coverage is 91.86047% with 7 lines in your changes missing coverage. Please review.

Project coverage is 72.11%. Comparing base (94a2009) to head (bd76bbe).
Report is 2 commits behind head on main.

Files with missing lines	Patch %	Lines
crates/core/src/operations/load_cdf.rs	87.50%	2 Missing and 2 partials ⚠️
crates/core/src/delta_datafusion/expr.rs	66.66%	0 Missing and 1 partial ⚠️
crates/core/src/delta_datafusion/mod.rs	97.50%	1 Missing ⚠️
python/src/lib.rs	0.00%	1 Missing ⚠️

Additional details and impacted files

@@           Coverage Diff           @@
##             main    #3261   +/-   ##
=======================================
  Coverage   72.11%   72.11%           
=======================================
  Files         143      143           
  Lines       45530    45545   +15     
  Branches    45530    45545   +15     
=======================================
+ Hits        32833    32846   +13     
- Misses      10618    10620    +2     
  Partials     2079     2079

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

alamb · 2025-02-28T19:13:50Z

🤔 there are some python failures here. Seeing if I can figure out what is going on

https://github.com/delta-io/delta-rs/actions/runs/13592071811/job/38008047004?pr=3261

FAILED tests/test_cdf.py::test_read_cdf_partitioned_projection - pyo3_runtime.PanicException: index out of bounds: the len is 3 but the index is 3
FAILED tests/test_cdf.py::test_read_cdf_last_version - pyo3_runtime.PanicException: index out of bounds: the len is 2 but the index is 2
==== 2 failed, 495 passed, 7 skipped, 35 deselected, 95 warnings in 40.15s =====

alamb · 2025-02-28T22:00:16Z

🤔 there are some python failures here. Seeing if I can figure out what is going on

https://github.com/delta-io/delta-rs/actions/runs/13592071811/job/38008047004?pr=3261
FAILED tests/test_cdf.py::test_read_cdf_partitioned_projection - pyo3_runtime.PanicException: index out of bounds: the len is 3 but the index is 3
FAILED tests/test_cdf.py::test_read_cdf_last_version - pyo3_runtime.PanicException: index out of bounds: the len is 2 but the index is 2
==== 2 failed, 495 passed, 7 skipped, 35 deselected, 95 warnings in 40.15s =====

Could someone tell me how to run these tests locally or give me a rust stack trace? I don't really understand what is failing here

blaginin · 2025-02-28T22:34:41Z

cd python
make develop
RUST_BACKTRACE=1 uv run pytest tests/test_cdf.py -s -k "test_read_cdf_partitioned_projection"

trace.txt

alamb requested review from wjones127, roeap, rtyler, hntd187 and ion-elgreco as code owners February 24, 2025 19:29

alamb marked this pull request as draft February 24, 2025 19:29

github-actions bot added the binding/rust Issues for the Rust crate label Feb 24, 2025

alamb mentioned this pull request Feb 24, 2025

Release DataFusion 46.0.0 apache/datafusion#14123

Open

25 tasks

alamb changed the title ~~core: upgrade to DataFusion 46.0.0 (WORK IN PROGRESS)~~ chore: upgrade to DataFusion 46.0.0 (WORK IN PROGRESS) Feb 24, 2025

ion-elgreco reviewed Feb 24, 2025

View reviewed changes

alamb mentioned this pull request Feb 24, 2025

chore: fix some compiler warnings #3262

Merged

This was referenced Feb 25, 2025

chore: fix clippy warnings on main #3264

Merged

chore: update tests to use Column::new and other expr_fn functions #3265

Merged

chore: use builder API to create FileScanConfig #3266

Merged

alamb mentioned this pull request Feb 25, 2025

Implement builder style API for ParserOptions apache/datafusion#14879

Closed

alamb force-pushed the alamb/df_46 branch from ce87ed1 to 2036ef1 Compare February 25, 2025 21:36

alamb force-pushed the alamb/df_46 branch from 2036ef1 to 47ea65c Compare February 26, 2025 12:26

github-actions bot added the binding/python Issues for the Python package label Feb 26, 2025

alamb commented Feb 26, 2025

View reviewed changes

This was referenced Feb 26, 2025

Rename DataSource and FileSource fields for consistency apache/datafusion#14898

Merged

Error projecting statistics in DataSourceExec apache/datafusion#14905

Closed

alamb force-pushed the alamb/df_46 branch 2 times, most recently from d300241 to c9b1903 Compare February 28, 2025 15:57

chore: Update to DataFusion 46.0.0, update for API chanages

1fb0b84

Signed-off-by: Andrew Lamb <[email protected]>

alamb force-pushed the alamb/df_46 branch from c9b1903 to 1fb0b84 Compare February 28, 2025 15:58

alamb commented Feb 28, 2025

View reviewed changes

fix: update builder rust version

bd76bbe

Signed-off-by: Andrew Lamb <[email protected]>

alamb commented Feb 28, 2025

View reviewed changes

alamb mentioned this pull request Feb 28, 2025

chore: remove unused dev dependency on home #3277

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

chore: upgrade to DataFusion 46.0.0 (WORK IN PROGRESS) #3261

chore: upgrade to DataFusion 46.0.0 (WORK IN PROGRESS) #3261

alamb commented Feb 24, 2025

github-actions bot commented Feb 24, 2025

ion-elgreco Feb 24, 2025

alamb Feb 24, 2025

ion-elgreco Feb 24, 2025

alamb Feb 25, 2025

alamb commented Feb 24, 2025

alamb commented Feb 24, 2025

alamb commented Feb 25, 2025

alamb Feb 26, 2025

alamb commented Feb 26, 2025 •

edited

Loading

alamb Feb 28, 2025

alamb Feb 28, 2025

alamb commented Feb 28, 2025

codecov bot commented Feb 28, 2025

alamb commented Feb 28, 2025

alamb commented Feb 28, 2025

blaginin commented Feb 28, 2025

chore: upgrade to DataFusion 46.0.0 (WORK IN PROGRESS) #3261

Are you sure you want to change the base?

chore: upgrade to DataFusion 46.0.0 (WORK IN PROGRESS) #3261

Conversation

alamb commented Feb 24, 2025

Description

Related Issue(s)

Documentation

github-actions bot commented Feb 24, 2025

ion-elgreco Feb 24, 2025

Choose a reason for hiding this comment

alamb Feb 24, 2025

Choose a reason for hiding this comment

ion-elgreco Feb 24, 2025

Choose a reason for hiding this comment

alamb Feb 25, 2025

Choose a reason for hiding this comment

alamb commented Feb 24, 2025

alamb commented Feb 24, 2025

alamb commented Feb 25, 2025

alamb Feb 26, 2025

Choose a reason for hiding this comment

alamb commented Feb 26, 2025 • edited Loading

alamb Feb 28, 2025

Choose a reason for hiding this comment

alamb Feb 28, 2025

Choose a reason for hiding this comment

alamb commented Feb 28, 2025

codecov bot commented Feb 28, 2025

Codecov Report

alamb commented Feb 28, 2025

alamb commented Feb 28, 2025

blaginin commented Feb 28, 2025

alamb commented Feb 26, 2025 •

edited

Loading