Skip to content

Commit

Permalink
Merge pull request #83 from Scienfitz/main
Browse files Browse the repository at this point in the history
fix: saving dataset via optional pandas / polars
  • Loading branch information
jonas-w authored Nov 11, 2024
2 parents 1cfdd81 + d31c809 commit 6d2ab03
Show file tree
Hide file tree
Showing 2 changed files with 10 additions and 2 deletions.
6 changes: 6 additions & 0 deletions docs/changelog.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,11 @@ All notable changes to this project will be documented in this file.
The format is based on [Keep a Changelog],
and this project adheres to [Semantic Versioning].

## [2.1.11]

## Fixed
- `save_dataframe` incorrectly interacting with optional modules pandas or polars

## [2.1.10] - 2024-10-31

## Changed
Expand Down Expand Up @@ -306,6 +311,7 @@ and this project adheres to [Semantic Versioning].
[Keep a Changelog]: https://keepachangelog.com/en/1.0.0/
[Semantic Versioning]: https://semver.org/spec/v2.0.0.html

[2.1.11]: https://github.com/emdgroup/foundry-dev-tools/compare/v2.1.10...v2.1.11
[2.1.10]: https://github.com/emdgroup/foundry-dev-tools/compare/v2.1.9...v2.1.10
[2.1.9]: https://github.com/emdgroup/foundry-dev-tools/compare/v2.1.8...v2.1.9
[2.1.8]: https://github.com/emdgroup/foundry-dev-tools/compare/v2.1.7...v2.1.8
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -567,11 +567,13 @@ def save_dataframe(
folder = str(round(time.time() * 1000)) if transaction_type == "APPEND" else "spark"
parquet_compression = "snappy"

if not (pd.__fake__ or pl.__fake__) and isinstance(df, pd.DataFrame | pl.DataFrame):
use_pandas = not pd.__fake__ and isinstance(df, pd.DataFrame)
use_polars = not pl.__fake__ and isinstance(df, pl.DataFrame)
if use_pandas or use_polars:
buf = io.BytesIO()
schema_flavor = "spark"

if isinstance(df, pd.DataFrame):
if use_pandas:
# write pandas dataframe as parquet to buffer
df.to_parquet(
buf,
Expand Down

0 comments on commit 6d2ab03

Please sign in to comment.