You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
SELECT [col("lst").list.contains([true]).alias("check")] FROM
DF ["lst"]; PROJECT 1/1 COLUMNS
Log output
Issue description
If you use list.any/all and the filter on something else, the filtering step isn't pushed down to before the any/all call.
I also saw similar optimization problems around using these functions, for example filtering on list.any and then calling head should be quite fast (as the code should finish when 5 filtered rows are found), but can be very slow for large frames (as I think the filter is calculated on all rows).
Expected behavior
These optimizations should be used (list.any has slightly different behavior than list.contains(True), but not enough to explain this)
Checks
Reproducible example
prints
while
prints
Log output
Issue description
If you use
list.any/all
and the filter on something else, the filtering step isn't pushed down to before theany/all
call.I also saw similar optimization problems around using these functions, for example filtering on
list.any
and then callinghead
should be quite fast (as the code should finish when 5 filtered rows are found), but can be very slow for large frames (as I think the filter is calculated on all rows).Expected behavior
These optimizations should be used (
list.any
has slightly different behavior thanlist.contains(True)
, but not enough to explain this)Installed versions
The text was updated successfully, but these errors were encountered: