From bf6f83b3d228fb386f9b4b20c254fa58e2412660 Mon Sep 17 00:00:00 2001 From: Andy Grove Date: Fri, 16 Feb 2024 08:08:23 -0700 Subject: [PATCH] generate changelog --- datafusion/CHANGELOG.md | 1 + dev/changelog/36.0.0.md | 264 ++++++++++++++++++++++++++++++++++++++++ 2 files changed, 265 insertions(+) create mode 100644 dev/changelog/36.0.0.md diff --git a/datafusion/CHANGELOG.md b/datafusion/CHANGELOG.md index ae9da0e865e9..2d09782a3982 100644 --- a/datafusion/CHANGELOG.md +++ b/datafusion/CHANGELOG.md @@ -19,6 +19,7 @@ # Changelog +- [36.0.0](../dev/changelog/36.0.0.md) - [35.0.0](../dev/changelog/35.0.0.md) - [34.0.0](../dev/changelog/34.0.0.md) - [33.0.0](../dev/changelog/33.0.0.md) diff --git a/dev/changelog/36.0.0.md b/dev/changelog/36.0.0.md new file mode 100644 index 000000000000..86f6a380ceb0 --- /dev/null +++ b/dev/changelog/36.0.0.md @@ -0,0 +1,264 @@ + + +## [36.0.0](https://github.com/apache/arrow-datafusion/tree/36.0.0) (2024-02-16) + +[Full Changelog](https://github.com/apache/arrow-datafusion/compare/35.0.0...36.0.0) + +**Breaking changes:** + +- Deprecate make_scalar_function [#8878](https://github.com/apache/arrow-datafusion/pull/8878) (viirya) +- Change `Accumulator::evaluate` and `Accumulator::state` to take `&mut self` [#8925](https://github.com/apache/arrow-datafusion/pull/8925) (alamb) +- Rename `CatalogList` to `CatalogProviderList` [#9002](https://github.com/apache/arrow-datafusion/pull/9002) (comphead) +- Remove some recursive cloning from logical planning [#9050](https://github.com/apache/arrow-datafusion/pull/9050) (ozankabak) +- Support `FixedSizeList` type coercion [#8902](https://github.com/apache/arrow-datafusion/pull/8902) (Weijun-H) +- Add `ColumnarValue::values_to_arrays`, deprecate `columnar_values_to_array` [#9114](https://github.com/apache/arrow-datafusion/pull/9114) (alamb) + +**Performance related:** + +- Minor: Add new Extended ClickBench benchmark queries [#8950](https://github.com/apache/arrow-datafusion/pull/8950) (alamb) + +**Implemented enhancements:** + +- feat: support `stride` in `array_slice`, change indexes to be`1` based [#8829](https://github.com/apache/arrow-datafusion/pull/8829) (Weijun-H) +- feat: emitting partial join results in `HashJoinStream` [#8020](https://github.com/apache/arrow-datafusion/pull/8020) (korowa) +- feat:implement sql style 'ends_with' and 'instr' string function [#8862](https://github.com/apache/arrow-datafusion/pull/8862) (zy-kkk) +- feat: Support parquet bloom filter pruning for decimal128 [#8930](https://github.com/apache/arrow-datafusion/pull/8930) (Ted-Jiang) +- feat: Disable client console highlight by default [#9013](https://github.com/apache/arrow-datafusion/pull/9013) (comphead) +- feat: support the ergonomics of getting list slice with stride [#8946](https://github.com/apache/arrow-datafusion/pull/8946) (Weijun-H) +- feat: Parallel Arrow file format reading [#8897](https://github.com/apache/arrow-datafusion/pull/8897) (my-vegetable-has-exploded) +- feat: support array_reverse [#9023](https://github.com/apache/arrow-datafusion/pull/9023) (Weijun-H) +- feat: issue #8969 adding position function [#8988](https://github.com/apache/arrow-datafusion/pull/8988) (Lordworms) +- feat: support `LargeList` in `flatten` [#9110](https://github.com/apache/arrow-datafusion/pull/9110) (Weijun-H) +- feat: improve `make_date` performance [#9112](https://github.com/apache/arrow-datafusion/pull/9112) (r3stl355) +- feat: add github action to self-assign the issue [#9132](https://github.com/apache/arrow-datafusion/pull/9132) (r3stl355) +- feat: add ability to query the remote http(s) location directly in datafusion-cli [#9150](https://github.com/apache/arrow-datafusion/pull/9150) (r3stl355) +- feat: implement select directly from s3 and gcs locations in datafusion-cli [#9199](https://github.com/apache/arrow-datafusion/pull/9199) (r3stl355) +- feat: support block gzip for streams [#9175](https://github.com/apache/arrow-datafusion/pull/9175) (tshauck) + +**Fixed bugs:** + +- fix: recursive initialize method [#8937](https://github.com/apache/arrow-datafusion/pull/8937) (waynexia) +- fix: common_subexpr_eliminate rule should not apply to short-circuit expression [#8928](https://github.com/apache/arrow-datafusion/pull/8928) (haohuaijin) +- fix: issue #8922 make row group test more readable [#8941](https://github.com/apache/arrow-datafusion/pull/8941) (Lordworms) +- fix: allow placeholders to be substituted when coercible [#8977](https://github.com/apache/arrow-datafusion/pull/8977) (kallisti-dev) +- fix: unambiguously truncate time in date_trunc function [#9068](https://github.com/apache/arrow-datafusion/pull/9068) (mhilton) +- fix: schema metadata retrieval when listing parquet table [#9134](https://github.com/apache/arrow-datafusion/pull/9134) (brayanjuls) + +**Documentation updates:** + +- Prepare 35.0.0-rc1 [#8924](https://github.com/apache/arrow-datafusion/pull/8924) (andygrove) +- Update project links [#8954](https://github.com/apache/arrow-datafusion/pull/8954) (comphead) +- Document parallelism and thread scheduling in the architecture guide [#8986](https://github.com/apache/arrow-datafusion/pull/8986) (alamb) +- chore: fix license badge in README [#9008](https://github.com/apache/arrow-datafusion/pull/9008) (suyanhanx) +- docs: fix array_position docs [#9003](https://github.com/apache/arrow-datafusion/pull/9003) (tshauck) +- Docs: improve contributor guide to explain how to work with tickets [#8999](https://github.com/apache/arrow-datafusion/pull/8999) (alamb) +- Document minimum required rust version [#9071](https://github.com/apache/arrow-datafusion/pull/9071) (comphead) +- Minor: Add ParadeDB to the list of users [#9018](https://github.com/apache/arrow-datafusion/pull/9018) (alamb) +- Update minimum rust version to 1.72 [#8997](https://github.com/apache/arrow-datafusion/pull/8997) (alamb) +- docs: add docs and example showing how to get the expression data type [#9118](https://github.com/apache/arrow-datafusion/pull/9118) (r3stl355) +- chore: Fix incorrect comment in substrait consumer [#9123](https://github.com/apache/arrow-datafusion/pull/9123) (caicancai) +- Minor: Fix Self referential links in readme [#9119](https://github.com/apache/arrow-datafusion/pull/9119) (alamb) +- Examples link in catalogs.rs leads to a 404 [#9194](https://github.com/apache/arrow-datafusion/pull/9194) (Omega359) +- Create `datafusion-functions-array` crate and move `ArrayToString` function into it [#9113](https://github.com/apache/arrow-datafusion/pull/9113) (alamb) + +**Merged pull requests:** + +- Add hash_join_single_partition_threshold_rows config [#8720](https://github.com/apache/arrow-datafusion/pull/8720) (maruschin) +- Prepare 35.0.0-rc1 [#8924](https://github.com/apache/arrow-datafusion/pull/8924) (andygrove) +- feat: support `stride` in `array_slice`, change indexes to be`1` based [#8829](https://github.com/apache/arrow-datafusion/pull/8829) (Weijun-H) +- fix: recursive initialize method [#8937](https://github.com/apache/arrow-datafusion/pull/8937) (waynexia) +- Fix expr partial ord test [#8908](https://github.com/apache/arrow-datafusion/pull/8908) (mustafasrepo) +- Simplify windows builtin functions return type [#8920](https://github.com/apache/arrow-datafusion/pull/8920) (comphead) +- Fix handling of nested leaf columns in parallel parquet writer [#8923](https://github.com/apache/arrow-datafusion/pull/8923) (devinjdangelo) +- feat: emitting partial join results in `HashJoinStream` [#8020](https://github.com/apache/arrow-datafusion/pull/8020) (korowa) +- fix: common_subexpr_eliminate rule should not apply to short-circuit expression [#8928](https://github.com/apache/arrow-datafusion/pull/8928) (haohuaijin) +- Support GroupsAccumulator accumulator for udaf [#8892](https://github.com/apache/arrow-datafusion/pull/8892) (guojidan) +- test: Port tests in `partitioned_csv.rs` to sqllogictest [#8919](https://github.com/apache/arrow-datafusion/pull/8919) (simicd) +- [CI] Fix RUSTFLAGS [#8929](https://github.com/apache/arrow-datafusion/pull/8929) (Jefffrey) +- Minor: Update datafusion-cli README to explain why it is not in the w… [#8938](https://github.com/apache/arrow-datafusion/pull/8938) (alamb) +- Add syntax highlight to datafusion-cli [#8918](https://github.com/apache/arrow-datafusion/pull/8918) (trungda) +- Update substrait requirement from 0.22.1 to 0.23.0 [#8943](https://github.com/apache/arrow-datafusion/pull/8943) (dependabot[bot]) +- Deprecate make_scalar_function [#8878](https://github.com/apache/arrow-datafusion/pull/8878) (viirya) +- Update project links [#8954](https://github.com/apache/arrow-datafusion/pull/8954) (comphead) +- fix: issue #8922 make row group test more readable [#8941](https://github.com/apache/arrow-datafusion/pull/8941) (Lordworms) +- feat:implement sql style 'ends_with' and 'instr' string function [#8862](https://github.com/apache/arrow-datafusion/pull/8862) (zy-kkk) +- [MINOR]: Extract aggregate topk function to `aggregate_topk.slt` [#8948](https://github.com/apache/arrow-datafusion/pull/8948) (mustafasrepo) +- Combine multiple `IN` lists in `ExprSimplifier` [#8949](https://github.com/apache/arrow-datafusion/pull/8949) (jayzhan211) +- Fix clippy failures: error: use of deprecated function `functions::make_scalar_function [#8972](https://github.com/apache/arrow-datafusion/pull/8972) (alamb) +- feat: Support parquet bloom filter pruning for decimal128 [#8930](https://github.com/apache/arrow-datafusion/pull/8930) (Ted-Jiang) +- [MINOR]: Update create_window_expr to refer only input schema [#8945](https://github.com/apache/arrow-datafusion/pull/8945) (mustafasrepo) +- Don't error in simplify_expressions rule [#8957](https://github.com/apache/arrow-datafusion/pull/8957) (haohuaijin) +- Use .zip to avoid unwrap [#8956](https://github.com/apache/arrow-datafusion/pull/8956) (Luv-Ray) +- Change `Accumulator::evaluate` and `Accumulator::state` to take `&mut self` [#8925](https://github.com/apache/arrow-datafusion/pull/8925) (alamb) +- Enhance simplifier by adding Canonicalize [#8780](https://github.com/apache/arrow-datafusion/pull/8780) (yyy1000) +- Find the correct fields when using page filter on `struct` fields in parquet [#8848](https://github.com/apache/arrow-datafusion/pull/8848) (manoj-inukolunu) +- fix: allow placeholders to be substituted when coercible [#8977](https://github.com/apache/arrow-datafusion/pull/8977) (kallisti-dev) +- Minor: improve CatalogProvider documentation with rationale and info about remote catalogs [#8968](https://github.com/apache/arrow-datafusion/pull/8968) (alamb) +- Improve to_timestamp docs [#8981](https://github.com/apache/arrow-datafusion/pull/8981) (Omega359) +- Add helper function for processing scalar function input [#8962](https://github.com/apache/arrow-datafusion/pull/8962) (viirya) +- Fix optimize projections bug [#8960](https://github.com/apache/arrow-datafusion/pull/8960) (mustafasrepo) +- NOT operator not return internal error when args are not boolean value [#8982](https://github.com/apache/arrow-datafusion/pull/8982) (guojidan) +- Minor: Add new Extended ClickBench benchmark queries [#8950](https://github.com/apache/arrow-datafusion/pull/8950) (alamb) +- Minor: Add comments to MSRV CI check to help if it fails [#8995](https://github.com/apache/arrow-datafusion/pull/8995) (alamb) +- Minor: Document memory management design on `MemoryPool` [#8966](https://github.com/apache/arrow-datafusion/pull/8966) (alamb) +- Fix LEAD/LAG window functions when default value null [#8989](https://github.com/apache/arrow-datafusion/pull/8989) (comphead) +- Optimize MIN/MAX when relation is empty [#8940](https://github.com/apache/arrow-datafusion/pull/8940) (viirya) +- [task #8203] Port tests in joins.rs to sqllogictest [#8996](https://github.com/apache/arrow-datafusion/pull/8996) (Tangruilin) +- [task #8213]Port tests in select.rs to sqllogictest [#8967](https://github.com/apache/arrow-datafusion/pull/8967) (Tangruilin) +- test: Port (last) `repartition.rs` query to sqllogictest [#8936](https://github.com/apache/arrow-datafusion/pull/8936) (simicd) +- Update to sqlparser `0.42.0` [#9000](https://github.com/apache/arrow-datafusion/pull/9000) (alamb) +- [MINOR]: Fix Optimize Projections Bug [#8992](https://github.com/apache/arrow-datafusion/pull/8992) (mustafasrepo) +- Make Topk aggregate tests deterministic [#8998](https://github.com/apache/arrow-datafusion/pull/8998) (mustafasrepo) +- Add support for Postgres LIKE operators [#8894](https://github.com/apache/arrow-datafusion/pull/8894) (gruuya) +- bug: Datafusion doesn't respect case sensitive table references [#8964](https://github.com/apache/arrow-datafusion/pull/8964) (xhwhis) +- Document parallelism and thread scheduling in the architecture guide [#8986](https://github.com/apache/arrow-datafusion/pull/8986) (alamb) +- Fix None Projections in Projection Pushdown [#9005](https://github.com/apache/arrow-datafusion/pull/9005) (berkaysynnada) +- Lead and Lag window functions should support default value with datatype other than Int64 [#9001](https://github.com/apache/arrow-datafusion/pull/9001) (viirya) +- chore: fix license badge in README [#9008](https://github.com/apache/arrow-datafusion/pull/9008) (suyanhanx) +- Minor: fix: #9010 - Optimizer schema change assert error is incorrect [#9012](https://github.com/apache/arrow-datafusion/pull/9012) (curtisleefulton) +- docs: fix array_position docs [#9003](https://github.com/apache/arrow-datafusion/pull/9003) (tshauck) +- Rename `CatalogList` to `CatalogProviderList` [#9002](https://github.com/apache/arrow-datafusion/pull/9002) (comphead) +- Safeguard against potential inexact row count being smaller than exact null count [#9007](https://github.com/apache/arrow-datafusion/pull/9007) (gruuya) +- Recursive CTEs: Stage 3 - add execution support [#8840](https://github.com/apache/arrow-datafusion/pull/8840) (matthewgapp) +- sqllogictest: move the creation of the nan_table from Rust to slt [#9022](https://github.com/apache/arrow-datafusion/pull/9022) (jonahgao) +- TreeNode refactor code deduplication: Part 3 [#8817](https://github.com/apache/arrow-datafusion/pull/8817) (ozankabak) +- feat: Disable client console highlight by default [#9013](https://github.com/apache/arrow-datafusion/pull/9013) (comphead) +- [task #8917] Implement information_schema.schemata [#8993](https://github.com/apache/arrow-datafusion/pull/8993) (Tangruilin) +- Properly encode STRING_AGG, NTH_VALUE in physical plan protobufs [#9027](https://github.com/apache/arrow-datafusion/pull/9027) (scsmithr) +- [task #8201] Port tests in expr.rs to sqllogictest, finish the left c… [#9014](https://github.com/apache/arrow-datafusion/pull/9014) (Tangruilin) +- Fix the clippy error of use of deprecated method [#9034](https://github.com/apache/arrow-datafusion/pull/9034) (viirya) +- feat: support the ergonomics of getting list slice with stride [#8946](https://github.com/apache/arrow-datafusion/pull/8946) (Weijun-H) +- Cache common referred expression at the window input [#9009](https://github.com/apache/arrow-datafusion/pull/9009) (mustafasrepo) +- Optimize `COUNT( DISTINCT ...)` for strings (up to 9x faster) [#8849](https://github.com/apache/arrow-datafusion/pull/8849) (jayzhan211) +- feat: Parallel Arrow file format reading [#8897](https://github.com/apache/arrow-datafusion/pull/8897) (my-vegetable-has-exploded) +- Change remove from swap to shift in index map [#9049](https://github.com/apache/arrow-datafusion/pull/9049) (mustafasrepo) +- Relax join keys constraint from Column to any physical expression for physical join operators [#8991](https://github.com/apache/arrow-datafusion/pull/8991) (viirya) +- Minor: Improve memory helper trait documentation [#9025](https://github.com/apache/arrow-datafusion/pull/9025) (alamb) +- Docs: improve contributor guide to explain how to work with tickets [#8999](https://github.com/apache/arrow-datafusion/pull/8999) (alamb) +- fix issue where upper and lower functions only work correctly on ascii character [#9054](https://github.com/apache/arrow-datafusion/pull/9054) (Omega359) +- Minor: small updates to bench.sh [#9035](https://github.com/apache/arrow-datafusion/pull/9035) (kmitchener) +- Chore: explicitly list out all Expr types in TypeCoercionRewriter::mutate [#9038](https://github.com/apache/arrow-datafusion/pull/9038) (guojidan) +- Minor: improve scalar functions document [#9029](https://github.com/apache/arrow-datafusion/pull/9029) (Weijun-H) +- [MINOR] Alter a SHJ test for relaxing "on" condition [#9065](https://github.com/apache/arrow-datafusion/pull/9065) (metesynnada) +- Remove some recursive cloning from logical planning [#9050](https://github.com/apache/arrow-datafusion/pull/9050) (ozankabak) +- minor: remove useless macro [#8979](https://github.com/apache/arrow-datafusion/pull/8979) (jackwener) +- Causality Analysis for Builtin Window Functions [#9048](https://github.com/apache/arrow-datafusion/pull/9048) (mustafasrepo) +- Minor: add doc examples for RawTableAllocExt [#9059](https://github.com/apache/arrow-datafusion/pull/9059) (alamb) +- Update substrait requirement from 0.23.0 to 0.24.0 [#9067](https://github.com/apache/arrow-datafusion/pull/9067) (dependabot[bot]) +- Remove single_file_output option from FileSinkConfig and Copy statement [#9041](https://github.com/apache/arrow-datafusion/pull/9041) (yyy1000) +- Add a make_date function [#9040](https://github.com/apache/arrow-datafusion/pull/9040) (Omega359) +- Speedup `DFSchema::merge` using HashSet indices [#9020](https://github.com/apache/arrow-datafusion/pull/9020) (simonvandel) +- Document minimum required rust version [#9071](https://github.com/apache/arrow-datafusion/pull/9071) (comphead) +- Return proper number of expressions for nth_value_agg [#9044](https://github.com/apache/arrow-datafusion/pull/9044) (mustafasrepo) +- ScalarUDF with zero arguments should be provided with one null array as parameter [#9031](https://github.com/apache/arrow-datafusion/pull/9031) (viirya) +- Update strum requirement from 0.25.0 to 0.26.1 [#9046](https://github.com/apache/arrow-datafusion/pull/9046) (dependabot[bot]) +- Create `datafusion-functions` crate, extract encode and decode to [#8705](https://github.com/apache/arrow-datafusion/pull/8705) (alamb) +- Add documentation for streaming usecase [#9070](https://github.com/apache/arrow-datafusion/pull/9070) (mustafasrepo) +- fix: unambiguously truncate time in date_trunc function [#9068](https://github.com/apache/arrow-datafusion/pull/9068) (mhilton) +- feat: support array_reverse [#9023](https://github.com/apache/arrow-datafusion/pull/9023) (Weijun-H) +- prettier to_timestamp_invoke [#9078](https://github.com/apache/arrow-datafusion/pull/9078) (Tangruilin) +- Handle invalid types for negation [#9066](https://github.com/apache/arrow-datafusion/pull/9066) (trungda) +- Minor: reduce unwraps in datetime_expressions.rs [#9072](https://github.com/apache/arrow-datafusion/pull/9072) (alamb) +- Remove custom doubling strategy + add examples to `VecAllocEx` [#9058](https://github.com/apache/arrow-datafusion/pull/9058) (alamb) +- Split physical_plan_tpch into separate benchmarks [#9043](https://github.com/apache/arrow-datafusion/pull/9043) (simonvandel) +- Minor: Add ParadeDB to the list of users [#9018](https://github.com/apache/arrow-datafusion/pull/9018) (alamb) +- [MINOR]: Add check for unnecessary projection [#9079](https://github.com/apache/arrow-datafusion/pull/9079) (mustafasrepo) +- chore(placeholder): update error message and add tests [#9073](https://github.com/apache/arrow-datafusion/pull/9073) (appletreeisyellow) +- refer to #8781, convert the internal_err! in datetime_expression.rs to exec_err! [#9083](https://github.com/apache/arrow-datafusion/pull/9083) (Tangruilin) +- Add benchmarks for to_timestamp and make_date functions [#9086](https://github.com/apache/arrow-datafusion/pull/9086) (Omega359) +- chore: Clarify ParadeDB branding [#9088](https://github.com/apache/arrow-datafusion/pull/9088) (philippemnoel) +- doc: Add example how to include latest datafusion [#9076](https://github.com/apache/arrow-datafusion/pull/9076) (comphead) +- Update minimum rust version to 1.72 [#8997](https://github.com/apache/arrow-datafusion/pull/8997) (alamb) +- Fix typo in an error message [#9099](https://github.com/apache/arrow-datafusion/pull/9099) (AdamGS) +- Update InfluxDB links in Known Users section of documentation [#9092](https://github.com/apache/arrow-datafusion/pull/9092) (alamb) +- Support `FixedSizeList` type coercion [#8902](https://github.com/apache/arrow-datafusion/pull/8902) (Weijun-H) +- Improve Canonicalize API [#8983](https://github.com/apache/arrow-datafusion/pull/8983) (alamb) +- Update env_logger requirement from 0.10 to 0.11 [#8944](https://github.com/apache/arrow-datafusion/pull/8944) (dependabot[bot]) +- Split count_distinct.rs into separate modules [#9087](https://github.com/apache/arrow-datafusion/pull/9087) (alamb) +- Fix update_expr for projection pushdown [#9096](https://github.com/apache/arrow-datafusion/pull/9096) (viirya) +- Improve `InListSImplifier` -- add test, commend and avoid clones [#8971](https://github.com/apache/arrow-datafusion/pull/8971) (alamb) +- feat: issue #8969 adding position function [#8988](https://github.com/apache/arrow-datafusion/pull/8988) (Lordworms) +- Cleanup regex_expressions.rs to remove \_regexp_match function [#9107](https://github.com/apache/arrow-datafusion/pull/9107) (Omega359) +- Unnest with single expression [#9069](https://github.com/apache/arrow-datafusion/pull/9069) (jayzhan211) +- Minor: improve GroupsAccumulator and Accumulator documentation [#8963](https://github.com/apache/arrow-datafusion/pull/8963) (alamb) +- move InList related simplify to one place [#9037](https://github.com/apache/arrow-datafusion/pull/9037) (guojidan) +- docs: add docs and example showing how to get the expression data type [#9118](https://github.com/apache/arrow-datafusion/pull/9118) (r3stl355) +- Add http(s) support to the command line [#8753](https://github.com/apache/arrow-datafusion/pull/8753) (kcolford) +- Remove External Table Backwards Compatibility Options [#9105](https://github.com/apache/arrow-datafusion/pull/9105) (yyy1000) +- feat: support `LargeList` in `flatten` [#9110](https://github.com/apache/arrow-datafusion/pull/9110) (Weijun-H) +- feat: improve `make_date` performance [#9112](https://github.com/apache/arrow-datafusion/pull/9112) (r3stl355) +- Refactor min/max value update in Parquet statistics [#9120](https://github.com/apache/arrow-datafusion/pull/9120) (Weijun-H) +- chore: Fix incorrect comment in substrait consumer [#9123](https://github.com/apache/arrow-datafusion/pull/9123) (caicancai) +- Minor: Fix Self referential links in readme [#9119](https://github.com/apache/arrow-datafusion/pull/9119) (alamb) +- Add `ColumnarValue::values_to_arrays`, deprecate `columnar_values_to_array` [#9114](https://github.com/apache/arrow-datafusion/pull/9114) (alamb) +- Support Copy with Remote Object Stores in datafusion-cli [#9064](https://github.com/apache/arrow-datafusion/pull/9064) (manoj-inukolunu) +- Fix Dockerfile min rust version to 1.72 [#9135](https://github.com/apache/arrow-datafusion/pull/9135) (alamb) +- fix: schema metadata retrieval when listing parquet table [#9134](https://github.com/apache/arrow-datafusion/pull/9134) (brayanjuls) +- Update parse_protobuf_file_scan_config to remove any partition columns from the file_schema in FileScanConfig [#9126](https://github.com/apache/arrow-datafusion/pull/9126) (bcmcmill) +- feat: add github action to self-assign the issue [#9132](https://github.com/apache/arrow-datafusion/pull/9132) (r3stl355) +- Fix NULL values in FixedSizeList creation [#9141](https://github.com/apache/arrow-datafusion/pull/9141) (Weijun-H) +- Add `FunctionRegistry::register_udaf` and `FunctionRegistry::register_udwf` [#9075](https://github.com/apache/arrow-datafusion/pull/9075) (alamb) +- Change ScalarValue::Struct to ArrayRef [#7893](https://github.com/apache/arrow-datafusion/pull/7893) (jayzhan211) +- Support join filter for `SortMergeJoin` [#9080](https://github.com/apache/arrow-datafusion/pull/9080) (viirya) +- Typo in docstring [#9149](https://github.com/apache/arrow-datafusion/pull/9149) (tv42) +- RecordBatchReceiverStreamBuilder: don't stringify errors [#9155](https://github.com/apache/arrow-datafusion/pull/9155) (tv42) +- port position test to scalar [#9128](https://github.com/apache/arrow-datafusion/pull/9128) (Lordworms) +- Minor: Improve `DataFrame` docs, add examples [#9159](https://github.com/apache/arrow-datafusion/pull/9159) (alamb) +- feat: add ability to query the remote http(s) location directly in datafusion-cli [#9150](https://github.com/apache/arrow-datafusion/pull/9150) (r3stl355) +- Add `regexp_like, improve docs and examples for `regexp_match` [#9137](https://github.com/apache/arrow-datafusion/pull/9137) (Omega359) +- Partial Sort Plan Implementation [#9125](https://github.com/apache/arrow-datafusion/pull/9125) (ahmetenis) +- Update tonic requirement from 0.10 to 0.11 [#9176](https://github.com/apache/arrow-datafusion/pull/9176) (dependabot[bot]) +- minor: fix error message function naming [#9168](https://github.com/apache/arrow-datafusion/pull/9168) (comphead) +- Minor: Update `DataFrame::write_table` docs [#9169](https://github.com/apache/arrow-datafusion/pull/9169) (alamb) +- Improve PhysicalExpr documentation [#9180](https://github.com/apache/arrow-datafusion/pull/9180) (alamb) +- Fix sphinx warnings [#9142](https://github.com/apache/arrow-datafusion/pull/9142) (ongchi) +- Use concat to simplify Nested Scalar creation [#9174](https://github.com/apache/arrow-datafusion/pull/9174) (jayzhan211) +- Minor: Remove unecessary map_err [#9186](https://github.com/apache/arrow-datafusion/pull/9186) (alamb) +- Add example of using `PruningPredicate` to datafusion-examples [#9183](https://github.com/apache/arrow-datafusion/pull/9183) (alamb) +- Use prep_null_mask_filter to handle nulls in selection mask [#9163](https://github.com/apache/arrow-datafusion/pull/9163) (viirya) +- [Document] Adding UDF by impl ScalarUDFImpl [#9172](https://github.com/apache/arrow-datafusion/pull/9172) (yyy1000) +- Docs: Extend `PruningPredicate` with background and implementation info [#9184](https://github.com/apache/arrow-datafusion/pull/9184) (alamb) +- chore: make tokio a workspace dependency [#9187](https://github.com/apache/arrow-datafusion/pull/9187) (PsiACE) +- Examples link in catalogs.rs leads to a 404 [#9194](https://github.com/apache/arrow-datafusion/pull/9194) (Omega359) +- Add test pipeline for Mac aarch64 [#9191](https://github.com/apache/arrow-datafusion/pull/9191) (viirya) +- Add string aggregate grouping fuzz test, add `MemTable::with_sort_exprs` [#9190](https://github.com/apache/arrow-datafusion/pull/9190) (alamb) +- Create `datafusion-functions-array` crate and move `ArrayToString` function into it [#9113](https://github.com/apache/arrow-datafusion/pull/9113) (alamb) +- Add constant expression support to equivalence properties [#9198](https://github.com/apache/arrow-datafusion/pull/9198) (mustafasrepo) +- chore: update tpch-docker docker repository [#9204](https://github.com/apache/arrow-datafusion/pull/9204) (pmcgleenon) +- feat: implement select directly from s3 and gcs locations in datafusion-cli [#9199](https://github.com/apache/arrow-datafusion/pull/9199) (r3stl355) +- MINOR: Add "fs" feature to "tokio", fix "features" typo. [#9210](https://github.com/apache/arrow-datafusion/pull/9210) (mustafasrepo) +- Add `to_char` function implementation using chrono formats [#9181](https://github.com/apache/arrow-datafusion/pull/9181) (Omega359) +- Add `SessionContext::read_batches` [#9197](https://github.com/apache/arrow-datafusion/pull/9197) (Lordworms) +- feat: support block gzip for streams [#9175](https://github.com/apache/arrow-datafusion/pull/9175) (tshauck) +- chore(pruning): Support `IS NOT NULL` predicates in `PruningPredicate` [#9208](https://github.com/apache/arrow-datafusion/pull/9208) (appletreeisyellow) +- Add cargo audit CI [#9182](https://github.com/apache/arrow-datafusion/pull/9182) (ongchi) +- Move `nullif` and `isnan` to datafusion-functions [#9216](https://github.com/apache/arrow-datafusion/pull/9216) (alamb) +- Bugfix - Projection Removal Conditions [#9215](https://github.com/apache/arrow-datafusion/pull/9215) (berkaysynnada) +- Partitioning fixes [#9207](https://github.com/apache/arrow-datafusion/pull/9207) (esheppa) +- Return an error when a column does not exist in window function [#9202](https://github.com/apache/arrow-datafusion/pull/9202) (PhVHoang) +- Revert "chore(pruning): Support `IS NOT NULL` predicates in `PruningPredicate` (#9208)" [#9232](https://github.com/apache/arrow-datafusion/pull/9232) (appletreeisyellow) +- Improve documentation on how to build `ScalarValue::Struct` and add `ScalarStructBuilder` [#9229](https://github.com/apache/arrow-datafusion/pull/9229) (alamb) +- Minor: improve Display of output ordering of `StreamTableExec` [#9225](https://github.com/apache/arrow-datafusion/pull/9225) (mustafasrepo) +- Support compute return types from argument values (not just their DataTypes) [#8985](https://github.com/apache/arrow-datafusion/pull/8985) (yyy1000) +- Dont call multiunzip when no stats [#9220](https://github.com/apache/arrow-datafusion/pull/9220) (matthewmturner) +- Use setup-macos-aarch64-builder for aarch64 CI pipeline [#9242](https://github.com/apache/arrow-datafusion/pull/9242) (viirya) +- GROUP-BY prioritizes input columns in case of ambiguity [#9228](https://github.com/apache/arrow-datafusion/pull/9228) (jonahgao) +- Minor: chore: improve catalog test in mod.rs [#9244](https://github.com/apache/arrow-datafusion/pull/9244) (caicancai) +- Add example for `ScalarStructBuilder::new_null`, fix display for `null` `ScalarValue::Struct` [#9238](https://github.com/apache/arrow-datafusion/pull/9238) (alamb)