Use arrow IPC Stream format for spill files #14868
Open
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
davidhewitt
changed the title
davidhewitt
force-pushed
the
dh/spill-stream
branch
from
cc413d5 to
7ac9fdf
Compare
|
Maybe it's better to add tests to cover your issue? |
|
Good point, I will add them probably tomorrow 👍 |
2010YOUY01
reviewed
Feb 26, 2025
davidhewitt
force-pushed
the
dh/spill-stream
branch
from
7ac9fdf to
f904b63
Compare
|
Rebased on main & test pushed 👍 |
davidhewitt
commented
Feb 27, 2025
| } | ||
|
|
||
| #[test] | ||
| fn test_batch_spill_and_read_dictionary_arrays() -> Result<()> { |
There was a problem hiding this comment.
I confirmed that this test fails on main with the error in #4658
|
#14078 looks like a similar problem but I suspect that the IPC Stream format is not much different in cost to the file format. At least with the test added here if the file format is changed again in the future, dictionary arrays will not regress. |
|
Thanks @davidhewitt I think this PR is good, please check the clippy |
comphead
approved these changes
Feb 27, 2025
|
Thanks! |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
Rationale for this change
The IPC Stream format allows for dictionary replacement, unlike the IPC File format. As per #4658 (comment) the File format does not offer advantages for the spill use case.
What changes are included in this PR?
Replaced the functionality in spilled sorts to write the IPC Stream format, instead of the IPC File format.
Are these changes tested?
Covered by existing tests of spill, I adapted these where necessary.
Are there any user-facing changes?
The code change is internal only; if for some reason users were inspecting the contents of spilled files without using the datafusion APIs to read them, they will find the format has changed.