Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support bounds evaluation for temporal data types #14523

Open
wants to merge 5 commits into
base: main
Choose a base branch
from

Conversation

ch-sc
Copy link
Contributor

@ch-sc ch-sc commented Feb 6, 2025

Which issue does this PR close?

As discussed in 14237 temporal data types should be supported in bounds evaluation.

Rationale for this change

We want to extend the number of expressions that can evaluate bounds to calculate statistics and enable better/more query plan optimisations.

What changes are included in this PR?

  • Additional data types are supported: Timestamp, Date, Time, Duration, Interval
  • Additional binary operators are supported: NotEq, IsDistinctFrom, IsNotDistinctFrom
  • Whether an expression supports bounds evaluation or not is now part of the PhysicalExpr trait and therefore decided by every expression type itself. This will also enable UDFs to implement bounds evaluation.

Are these changes tested?

yes

Are there any user-facing changes?

PhysicalExpr and ScalarUDFImpl change, but both provide default implementations for additional functions.

@github-actions github-actions bot added logical-expr Logical plan and expressions physical-expr Physical Expressions optimizer Optimizer rules core Core DataFusion crate sqllogictest SQL Logic Tests (.slt) common Related to common crate labels Feb 6, 2025
@github-actions github-actions bot removed the sqllogictest SQL Logic Tests (.slt) label Feb 6, 2025
@berkaysynnada
Copy link
Contributor

Thank you @ch-sc for working on this. When you need a review, I can do that if you ping me.

@ch-sc
Copy link
Contributor Author

ch-sc commented Feb 7, 2025

Thanks @berkaysynnada! Yeah I'd like to get a review.

There is still a bug, though, that I don't understand yet. Sometimes the sort operator gets removed as can be seen in the repartition_scan.slt test.

@berkaysynnada
Copy link
Contributor

Sometimes the sort operator gets removed as can be seen in the repartition_scan.slt test.

Seems like you're losing the order requirement somehow

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
common Related to common crate core Core DataFusion crate logical-expr Logical plan and expressions optimizer Optimizer rules physical-expr Physical Expressions
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Add support for filter selectivity for more expression types
2 participants