Skip to content
This repository has been archived by the owner on Sep 30, 2024. It is now read-only.

Parquet files not found with S3 URI #429

Closed
esheehan-gsl opened this issue Oct 30, 2023 · 0 comments · Fixed by #430
Closed

Parquet files not found with S3 URI #429

esheehan-gsl opened this issue Oct 30, 2023 · 0 comments · Fixed by #430
Assignees
Labels
bug Something isn't working
Milestone

Comments

@esheehan-gsl
Copy link
Contributor

Describe the bug

Historical data for model runs does not load in production, but it does load in development. This appears to be an issue with the file URI handling with local files vs S3.

Based on the stack trace (below), it seems like the issue may be that a / is being stripped from the environment variable when Pandas attempts to load the file, so it’s loading s3:/ instead of s3://.

To Reproduce

Steps to reproduce the behavior:

  1. Go to https://unified-graphics.noaa.gov/
  2. Select a background, initialization time, and a scalar variable (ps, q, or t)
  3. Click Load
  4. Histogram and map load, but the time series remains empty

Expected behavior

Time series data should load.

Additional context

[2023-10-30 16:52:24,346] ERROR in routes: Diagnostic file not found
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/flask/app.py", line 867, in full_dispatch_request
    rv = self.dispatch_request()
  File "/usr/local/lib/python3.10/site-packages/flask/app.py", line 852, in dispatch_request
    return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args)
  File "/usr/src/app/src/unified_graphics/routes.py", line 141, in history
    data = diag.history(
  File "/usr/src/app/src/unified_graphics/diag.py", line 473, in history
    df = pd.read_parquet(
  File "/usr/local/lib/python3.10/site-packages/pandas/io/parquet.py", line 670, in read_parquet
    return impl.read(
  File "/usr/local/lib/python3.10/site-packages/pandas/io/parquet.py", line 265, in read
    path_or_handle, handles, filesystem = _get_path_or_handle(
  File "/usr/local/lib/python3.10/site-packages/pandas/io/parquet.py", line 139, in _get_path_or_handle
    handles = get_handle(
  File "/usr/local/lib/python3.10/site-packages/pandas/io/common.py", line 872, in get_handle
    handle = open(handle, ioargs.mode)
FileNotFoundError: [Errno 2] No such file or directory: 's3:/osti-modeling-dev-rtma-vis-prod/RTMA_HRRR_WCOSS_CONUS_REALTIME/t'
@esheehan-gsl esheehan-gsl added the bug Something isn't working label Oct 30, 2023
@esheehan-gsl esheehan-gsl added this to the Cycle 2023.4 milestone Oct 30, 2023
@esheehan-gsl esheehan-gsl self-assigned this Oct 30, 2023
@esheehan-gsl esheehan-gsl linked a pull request Oct 30, 2023 that will close this issue
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
bug Something isn't working
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant