Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

BUG: datetime64 column type fails parquet round trip #60774

Open
2 of 3 tasks
bmwilly opened this issue Jan 23, 2025 · 2 comments
Open
2 of 3 tasks

BUG: datetime64 column type fails parquet round trip #60774

bmwilly opened this issue Jan 23, 2025 · 2 comments
Labels
Bug Datetime Datetime data dtype IO Parquet parquet, feather

Comments

@bmwilly
Copy link

bmwilly commented Jan 23, 2025

Pandas version checks

  • I have checked that this issue has not already been reported.

  • I have confirmed this bug exists on the latest version of pandas.

  • I have confirmed this bug exists on the main branch of pandas.

Reproducible Example

df = pd.DataFrame(
    {
        "ts": pd.to_datetime(
            ["2021-01-01 00:00:00", "2021-01-01 00:00:01", "2021-01-01 00:00:02"]
        ).astype("datetime64[s]"),
    }
)

df.to_parquet("/tmp/df.parquet")
df2 = pd.read_parquet("/tmp/df.parquet")

Issue Description

Then

df.dtypes

gives

ts    datetime64[s]
dtype: object

but

df2.dtypes

gives

ts    datetime64[ms]
dtype: object

Expected Behavior

I would expect

df2.dtypes

to give

ts    datetime64[s]
dtype: object

Installed Versions

INSTALLED VERSIONS
------------------
commit                : 0691c5cf90477d3503834d983f69350f250a6ff7
python                : 3.11.10
python-bits           : 64
OS                    : Darwin
OS-release            : 24.2.0
Version               : Darwin Kernel Version 24.2.0: Fri Dec  6 19:02:41 PST 2024; root:xnu-11215.61.5~2/RELEASE_ARM64_T6030
machine               : arm64
processor             : arm
byteorder             : little
LC_ALL                : en_US.UTF-8
LANG                  : en_US.UTF-8
LOCALE                : en_US.UTF-8

pandas                : 2.2.3
numpy                 : 1.24.4
pytz                  : 2024.2
dateutil              : 2.9.0.post0
pip                   : 24.2
Cython                : 3.0.11
sphinx                : None
IPython               : 8.31.0
adbc-driver-postgresql: None
adbc-driver-sqlite    : None
bs4                   : 4.12.3
blosc                 : None
bottleneck            : 1.3.7
dataframe-api-compat  : None
fastparquet           : None
fsspec                : 2023.12.2
html5lib              : 1.1
hypothesis            : 6.123.17
gcsfs                 : None
jinja2                : 3.1.5
lxml.etree            : 5.3.0
matplotlib            : 3.8.4
numba                 : 0.60.0
numexpr               : 2.10.2
odfpy                 : None
openpyxl              : 3.1.5
pandas_gbq            : None
psycopg2              : 2.9.10
pymysql               : None
pyarrow               : 16.1.0
pyreadstat            : None
pytest                : 7.4.4
python-calamine       : None
pyxlsb                : None
s3fs                  : 2023.12.2
scipy                 : 1.13.0
sqlalchemy            : 2.0.36
tables                : 3.9.2
tabulate              : 0.9.0
xarray                : 2025.1.1
xlrd                  : 2.0.1
xlsxwriter            : 3.2.0
zstandard             : 0.23.0
tzdata                : 2024.2
qtpy                  : None
pyqt5                 : None
@bmwilly bmwilly added Bug Needs Triage Issue that has not been reviewed by a pandas team member labels Jan 23, 2025
@rhshadrach rhshadrach added Datetime Datetime data dtype IO Parquet parquet, feather and removed Needs Triage Issue that has not been reviewed by a pandas team member labels Jan 23, 2025
@rhshadrach
Copy link
Member

Thanks for the report! Further investigations and PRs to fix are welcome.

@snitish
Copy link
Contributor

snitish commented Jan 24, 2025

This seems to be a known issue in pyarrow. TLDR: arrow internally uses millis, micros and nanos as unit. We could always re-cast it to the right unit downstream in pd.read_parquet using the pandas metadata, but it's up for debate whether this fix should be made here or on pyarrow side.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Bug Datetime Datetime data dtype IO Parquet parquet, feather
Projects
None yet
Development

No branches or pull requests

3 participants