Release arrow-rs / parquet minor version 54.2.0 (~Mar~ Feb 2025)

[alamb]

Is your feature request related to a problem or challenge? Please describe what you are trying to do.

Our release schedule is here https://github.com/apache/arrow-rs?tab=readme-ov-file#release-versioning-and-schedule

• We released 54.1.0 in Jan/Feb 2025 Release arrow-rs / parquet minor version 54.1.0 (Jan 2025) #6929

Items to fix before the release:

• Verify with downstream DataFusion tests: WIP: Update to arrow/parquet 54.2.0 datafusion#14628

Wishlist Items

• Experimental parquet decoder with first-class selection pushdown support #6921 from @XiangpengHao
• Improve Arrow-IPC performance by avoiding Unsafe Unchecked IPC Read RecordBatch #3287 (from @totoroyyb and @alamb )

Describe the solution you'd like

Create a release

• Release notes and bump version: Update version to 54.2.0 and add CHANGELOG #7110
• Release Candidate + Vote thread: https://lists.apache.org/thread/my880mq37gjsyzx7sc37wfdsbcsx1zl3
• Release to crates.io / approval

Describe alternatives you've considered

Additional context

[alamb]

Realistically I can't do both this release and the object store on the same week

• Release object_store 0.12.0 (API breaking) Around Feb 30 2025 #6903

[alamb]

We have discovered a regression

• [Regression in 54.0.0]. Decimal cast to smaller precision gives invalid (off-by-one) result in some cases #7069

@Blizzara has requested a release with this fix

• fix: issue introduced in #6833 - less than equal check for scale in decimal conversion #7070 (comment)

I therefore plan to make a 54.2.0 release later this week

[alamb]

Since main is still the 54.x line and the fix has already been merged

• [Regression in 54.0.0]. Decimal cast to smaller precision gives invalid (off-by-one) result in some cases #7069

I'll plan to make a release directly from main

FYI @tustvold and @Jefffrey @etseidl

[alamb]

I tested the pre-release update of 54.2.0 with DataFusion

• WIP: Update to arrow/parquet 54.2.0 datafusion#14628

It turns out that arrow has rand in the public API as part of the bench_util and I got a few compile errors like

error[E0599]: the method `gen` exists for struct `StdRng`, but its trait bounds were not satisfied
  --> datafusion/functions-aggregate/benches/array_agg.rs:64:27
   |
64 |         let is_null = rng.gen::<f32>() < null_density;
   |                           ^^^ method cannot be called on `StdRng` due to unsatisfied trait bounds
   |
  ::: /Users/andrewlamb/.cargo/registry/src/index.crates.io-6f17d22bba15001f/rand-0.9.0/src/rngs/std.rs:70:1
   |
70 | pub struct StdRng(Rng);
   | ----------------- doesn't satisfy `rand::rngs::std::StdRng: rand::RngCore` or `rand::rngs::std::StdRng: rand::Rng`
   |
   = note: the following trait bounds were not satisfied:
           `rand::rngs::std::StdRng: rand::RngCore`
           which is required by `rand::rngs::std::StdRng: rand::Rng`
   = help: items from traits can only be used if the trait is in scope
help: trait `Rng` which provides `gen` is implemented but not in scope; perhaps you want to import it
   |
18 + use rand::rng::Rng;

There are two potential solutions I can think of

1. (the nicest) revert the upgrade to rand
2. (the easiest) Publicly export rand so the correct version can be used in the API

I am going to make a PR for 2

[tustvold]

I am going to make a PR for 2

Won't that still be a breaking change?

[alamb]

I am going to make a PR for 2

Won't that still be a breaking change?

Yes

[tustvold]

In which case it can't be released as 54.2.0?

Edit: I don't see a way to avoid reverting #7045

[alamb]

In which case it can't be released as 54.2.0?

Edit: I don't see a way to avoid reverting #7045

Yes I suppose you are right -- I will prepare some PRs to do so

[alamb]

Here is a revert PR:

• Revert "Update rand requirement from 0.8 to 0.9 (#7045)" #7125

[alamb]

Ok, I reverted the rand upgrade and verified the datafusion upgrade went smoothly:

• WIP: Update to arrow/parquet 54.2.0 datafusion#14628

[alamb]

I am working on updating the release notes and will prepare an RC

[alamb]

I have created a release candidate and begun the voting:

• https://lists.apache.org/thread/my880mq37gjsyzx7sc37wfdsbcsx1zl3