Sflux pyschism os error

[Adarshvb000]

I was trying to download surface forcing datas from era5 using pyschism package. It was workng all fine. but im getting an error as below
File "/home/schism/Desktop/sflux.py", line 16, in
er.write(outdir=outdir, start_date=startdate, rnday=rnday, air=True, rad=True, prc=True, bbox=bbox, output_interval=interval, overwrite=True)
File "/home/schism/miniforge3/envs/pyschism/lib/python3.9/site-packages/pyschism/forcing/nws/nws2/era5.py", line 307, in write
nx_grid, ny_grid = self.inventory.xy_grid()
File "/home/schism/miniforge3/envs/pyschism/lib/python3.9/site-packages/pyschism/forcing/nws/nws2/era5.py", line 94, in xy_grid
return np.meshgrid(self.lon, self.lat)
File "/home/schism/miniforge3/envs/pyschism/lib/python3.9/site-packages/pyschism/forcing/nws/nws2/era5.py", line 80, in lon
self._lon = Dataset(self.files[0]).variables['longitude'][:]
File "src/netCDF4/_netCDF4.pyx", line 2521, in netCDF4._netCDF4.Dataset.init
File "src/netCDF4/_netCDF4.pyx", line 2158, in netCDF4._netCDF4._ensure_nc_success
OSError: [Errno -51] NetCDF: Unknown file format: '/tmp/tmpwoys9n50/era5_20090910.nc'
Why is it so?

[SorooshMani-NOAA]

@Adarshvb000 this seems like an issue related to corrupted nc file. Does this consistently happen for a given date or it randomly happens and goes away if you rerun? If it's the latter it's most likely due to a connection issue where the file is not fully downloaded and later it fails to read it. However if it happens consistently for a given date please note what date do you get this error for. Thanks!

[Adarshvb000]

@Adarshvb000 this seems like an issue related to corrupted nc file. Does this consistently happen for a given date or it randomly happens and goes away if you rerun? If it's the latter it's most likely due to a connection issue where the file is not fully downloaded and later it fails to read it. However if it happens consistently for a given date please note what date do you get this error for. Thanks!

Dear Soroosh Mani,

I am experiencing an error with the process regardless of the dates I use. I even tried setting up the package on my laptop, but I am still encountering the same issue.

I have attached screenshots for your reference. The first image shows that the data download begins, but it sometimes stops at 50%, other times at 0%, before ultimately showing the error (as seen in the second image).

This process was working perfectly fine a month ago. I have tried using three different networks, but the issue remains unresolved.

Kindly assist in resolving this issue.

[Adarshvb000]

[SorooshMani-NOAA]

On the Copernicus website I see this:

Maybe the issue is due to their recent update ... In the warnings you get it also says that the old credentials won't work ... I'm not sure if it's enough that you just get the credentials or PySCHISM needs to also update anything. I haven't recently used Era5, @feiye-vims have you checked recently to see if it works fine or not?

[feiye-vims]

@SorooshMani-NOAA I used it to download era5 in August, and it worked okay then.

Breanna ([email protected]) and Felicio did it more recently in November. Breanna told me it could be slow if you download large amount of data, but she didn't mention anything about the corrupt nc. Maybe you can email them for more details.

[SorooshMani-NOAA]

@Adarshvb000 how large is your date range? Can you try getting week by week or something like that to see if it goes through? In any case the website says disruptions are expected ... so it might be that the servers have some issues right now! @felicio93 When did you last download Era5? Have you experienced this?

[felicio93]

Not that long ago, maybe a few weeks. Let me try it again here and see if I get an error.

[felicio93]

I am getting a certificate error, it seems like 7 days ago here:

Please be aware that as initially published in this topic, the cds-beta URL was planned to be discontinued and users were advised to update the URL in their .cdsapirc file by 20th October 2024.

We can confirm that the cds-beta URL is now discontinued. And this explains why some users are getting some SSL: CERTIFICATE_VERIFY_FAILED errors.

If you have not already done so, please update the URL in your .cdsapirc file as shown below:

url: https://cds.climate.copernicus.eu/api
key: xx-xx-xx-xx-xx

Alternatively, login here to check URL to use in .cdsapirc file.

Thank you

ECMWF Support
on behalf of the Data Stores team

could this be the reason?

[felicio93]

I updated the url from cds.beta... to the one above on my .cdsapirc and now I am getting the same error as @Adarshvb000.

[SorooshMani-NOAA]

Thanks for checking @felicio93 ... so it's related to CDS server update. It's still unclear though if the error is something that will be resolved (since disruptions are expected) or is there anything we need to change on PySCHISM.

[Adarshvb000]

The issue persists only with PySCHISM. I downloaded the data using CDSAPI without any problems, but with PySCHISM, it’s not working. I have tried durations of 2 days, 5 days, 10 days, and even 50 days, but all show a similar error. I’ve encountered the same issue on two different devices. Is anyone else successfully obtaining ERA5 datasets using PySCHISM?

[SorooshMani-NOAA]

Thanks for testing, we need to address this in PySCHISM then!