Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

add pregenerated files to speed up notebook run #35

Merged
merged 4 commits into from
Jan 10, 2025

Conversation

amsnyder
Copy link
Contributor

@amsnyder amsnyder commented Jan 10, 2025

This updates the Github pipeline's data pull to grab the following files from the OSN Pod:

  1. state data files (notebook_output_files/owrd_cache.nc and notebook_output_files/sf_efc.nc) that are generated in notebook 0c
  2. pop-up html files (notebook_output_files/Folium_maps) that are generated in notebook 1b

This allows us to bypass the creation of those files on the fly when running notebooks to speed the repo's automated testing.

This PR also fixes typo in the willamette directory name in data pull script. This will break the scripts until notebook 0a is updated by @ahaj to fix the typo there as well.

@amsnyder amsnyder marked this pull request as ready for review January 10, 2025 18:59
@amsnyder amsnyder requested a review from jmccreight January 10, 2025 19:25
@jmccreight
Copy link
Collaborator

it all looks good but there is one more misspelling to fix elsewhere

./pull_domain.py --name=williamette_river

@jmccreight
Copy link
Collaborator

Once that is fixed we'll see the time it takes to pull

@amsnyder
Copy link
Contributor Author

it all looks good but there is one more misspelling to fix elsewhere

./pull_domain.py --name=williamette_river

I just pushed this update. Thanks for catching it!

Copy link
Collaborator

@jmccreight jmccreight left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The pulls failed once on ubuntu, we'll have to watch and see how it goes. Takes 8 to 9 minutes to pull all the files.

@jmccreight jmccreight merged commit 971c1d2 into main Jan 10, 2025
0 of 3 checks passed
@amsnyder
Copy link
Contributor Author

The pulls failed once on ubuntu, we'll have to watch and see how it goes. Takes 8 to 9 minutes to pull all the files.

I saw that failure - it looked like it could be a rate limit or something on the OSN pod? I couldn't find any documentation on rate limits for the pod, but if it keeps happening, I'll ask their team if they know of any.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants