You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
xarray.open_mfdataset() + .load(scheduler="sync") hang with some large E3SM v3 datasets. The root cause was identified as being a regression between dask=2024.11.2 and dask=2024.12.0.
Describe the solution you'd like
Validate dask=2025.1.0 update fixes performance issue using MVCE from here
Perform complete v3 run with dask=2025.1.0
Update dask constraint to dask !=2024.12.0,!=2024.12.1 in dev.yml, ci.yml and pyproject.toml
Inform @xylar on status and if E3SM Unified can update its constraint
Describe alternatives you've considered
No response
Additional context
No response
The text was updated successfully, but these errors were encountered:
tomvothecoder
changed the title
[Feature]: Test e3sm_diags v3 full run with dask=2025.1.0 and update constraint
[Feature]: Test dask=2025.1.0 resolves performance issue with large v3 datasets and update constraint (if yes)
Jan 10, 2025
Is your feature request related to a problem?
Related to:
xarray.open_mfdataset()
and.load()
slow or hangs with 4D dataset (76 GB) after upgrading todask>=2024.12.0
pydata/xarray#9926xarray.open_mfdataset()
+.load(scheduler="sync"
) hang with some large E3SM v3 datasets. The root cause was identified as being a regression betweendask=2024.11.2
anddask=2024.12.0
.Describe the solution you'd like
dask=2025.1.0
update fixes performance issue using MVCE from heredask=2025.1.0
dask !=2024.12.0,!=2024.12.1
indev.yml
,ci.yml
andpyproject.toml
Describe alternatives you've considered
No response
Additional context
No response
The text was updated successfully, but these errors were encountered: