Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Expose all write options #1005

Open
matko opened this issue Jan 29, 2025 · 0 comments · May be fixed by #1006
Open

Expose all write options #1005

matko opened this issue Jan 29, 2025 · 0 comments · May be fixed by #1006
Labels
enhancement New feature or request

Comments

@matko
Copy link

matko commented Jan 29, 2025

Is your feature request related to a problem or challenge? Please describe what you are trying to do.
This issue is about improving the behavior of write_csv(...), write_json(...) and write_parquet(...).

Right now datafusion-python only exposes a handful of write options that are actually supported by datafusion. Notably, DataFrameWriteOptions is always just initialized with its defaults, which means the following features are not supported for any format:

  • partitioned (hive style) writes
  • sorted writes
  • insert option (though I suspect this is actually not supported anyway for any of the 3 exposed formats)
  • single file output (though in my experiments I've not been able to make this actually do anything in rust)

Furthermore, there's options for each format, of which only some are now exposed:

  • parquet: only global compression options are exposed, but the writer actually supports pretty fine grained column options that are now unusable from python.
  • csv: only header inclusion/exclusion is supported. The underlying writer supports a lot of options for setting up things like delimiters, quote style, what to do with nulls, etc.
  • json: No options are supported at all right now. The underlying writer supports compression.

Describe the solution you'd like
Expose all supported write options in datafusion-python. I think we should just support sending in dictionaries with these options in the names people would expect from the rust documentation. More important options could additionally be top-level keyword arguments in their own right, much like is already the case for parquet global compression.

Describe alternatives you've considered
One alternative is bypassing datafusion and using parquet directly from an arrow stream. This means not being able to work with object stores though, and even when object stores are not needed it's not very ergonomic.

Additional context
None

@matko matko added the enhancement New feature or request label Jan 29, 2025
@matko matko linked a pull request Jan 29, 2025 that will close this issue
4 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant