05-ExpandingTheWorkflow_Acquire.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "14cc6bc4-aab2-42fe-a100-ba5e78094c47",
   "metadata": {},
   "source": [
    "# Expanding the Workflow\n",
    "## Different Data\n",
    "Let's say we now want to get our streamflow data from the NWM Retrospective... maybe to compare with the USGS data.  Let's write a function to do that."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9172f771-d92c-4fa8-8d11-8afbcf911448",
   "metadata": {},
   "source": [
    "## Quick Sidebar: Zarr\n",
    "[Zarr](https://zarr.dev/) is an technology that allows for indexed access to large, online datasets. The National Water Model, and NOAA in general, have used ZARR extensively to make it easier to download portions of their datasets without having to download large netCDF files or large sets of netCDF files. You can think of Zarr working together with sets of netCDF files so that users can download just portions of the netCDF formatted dataset."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "65358b69-c6e5-434b-8ad1-1a1791b29755",
   "metadata": {},
   "source": [
    "Before we write the function, let's explore how to do this with some Jupyter cells"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "aaeb1f56-31e9-4533-8b15-ffb90d638eb0",
   "metadata": {},
   "outputs": [],
   "source": [
    "from s3fs import S3FileSystem, S3Map\n",
    "import xarray as xr\n",
    "\n",
    "# https://registry.opendata.aws/nwm-archive/\n",
    "bucket = 's3://noaa-nwm-retrospective-3-0-pds/CONUS/zarr'\n",
    "fs = S3FileSystem(anon=True)\n",
    "ds = xr.open_dataset(S3Map(f\"{bucket}/chrtout.zarr\", s3=fs), engine='zarr')\n",
    "ds\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "87210acb-0822-40ad-a8aa-45841cdf1dcb",
   "metadata": {},
   "source": [
    "## Quick Sidebar: xarray\n",
    "[xarray](https://xarray.dev/) is Python library standardizes access to a number of underlying array-like data formats including netCDF, Zarr, and GRIB. xarray is useful because it standardizes the code used to work with these file formats instead of using their own specific Python libraries, each of which has their own syntax and methodologies for manipulating the data. It also works seemlessly with a number of other common Python libraries including pandas, NumPy, and matplotlib for visualization."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "78a1c0c0-eb24-42ed-97b4-20ef0e33c1bb",
   "metadata": {},
   "outputs": [],
   "source": [
    "ds = ds.sel({'feature_id': 166176984})\n",
    "ds"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "55f8e2cb-4c09-456f-8a74-06fb49a342b5",
   "metadata": {},
   "outputs": [],
   "source": [
    "ds = ds.sel(time=slice('2016-06-01T00:00:00', '2016-11-01T00:00:00'))\n",
    "ds"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7acde736-b516-4c10-873e-aaa318a88bf1",
   "metadata": {},
   "outputs": [],
   "source": [
    "ds.get('streamflow').to_pandas()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "53dd610e-9a34-4893-a29f-c2dd0fcc3273",
   "metadata": {},
   "source": [
    "But, no name for the series like we had for the USGS data acquisition function, so let's give it an appropriate name so it meets the interface we defined between acquire / filter and manipulate."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1ed086d0-73eb-4f9b-bbc6-23edea3f24d4",
   "metadata": {},
   "outputs": [],
   "source": [
    "ds.get('streamflow').to_pandas().rename('streamflow (m^3 s^-1)')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bafb126a-bb80-43f9-bb0e-eb07d4eb1739",
   "metadata": {},
   "source": [
    "Now that we have all the pieces figured out, let's combine them all into a function and parameterize it with the same parameters as our other acquire / filter data function and test it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8de0d7d7-6d57-49fd-b97a-8422f4f57361",
   "metadata": {},
   "outputs": [],
   "source": [
    "def acquire_streamflow_nwm_retrospective(site, start, end):\n",
    "    bucket = 's3://noaa-nwm-retrospective-3-0-pds/CONUS/zarr'\n",
    "    fs = S3FileSystem(anon=True)\n",
    "    ds = xr.open_dataset(S3Map(f\"{bucket}/chrtout.zarr\", s3=fs), engine='zarr')\n",
    "\n",
    "    ds = ds.sel({'feature_id': site})\n",
    "    ds = ds.sel(time=slice(f'{start}T00:00:00', f'{end}T00:00:00'))\n",
    "\n",
    "    return ds.get('streamflow').to_pandas().rename('streamflow (m^3 s^-1)')\n",
    "\n",
    "acquire_streamflow_nwm_retrospective(166176984, '2016-06-01', '2016-11-01')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6d975f18-4269-40df-8cb9-d96c4a53efd4",
   "metadata": {},
   "source": [
    "## Putting our New Function Into the Workflow\n",
    "\n",
    "Now that we have a new \"Data Acquire / Filter\" function that matches the interface we've defined, let's drop it into our workflow...\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "19bcb2f1-a971-4dea-af39-be310d9fb380",
   "metadata": {},
   "outputs": [],
   "source": [
    "import dataretrieval.nwis as nwis\n",
    "from s3fs import S3FileSystem, S3Map\n",
    "import xarray as xr\n",
    "\n",
    "def acquire_streamflow_nwis_iv(site, start, end):\n",
    "    df = nwis.get_record(sites=site, service='iv', start=start, end=end, parameterCD='00060')\n",
    "    # https://help.waterdata.usgs.gov/parameter_cd?group_cd=PHY\n",
    "    return df['00060'].rename('streamflow (ft^3/s)')\n",
    "\n",
    "def acquire_streamflow_nwm_retrospective(site, start, end):\n",
    "    bucket = 's3://noaa-nwm-retrospective-3-0-pds/CONUS/zarr'\n",
    "    fs = S3FileSystem(anon=True)\n",
    "    ds = xr.open_dataset(S3Map(f\"{bucket}/chrtout.zarr\", s3=fs), engine='zarr')\n",
    "\n",
    "    ds = ds.sel({'feature_id': site})\n",
    "    ds = ds.sel(time=slice(f'{start}T00:00:00', f'{end}T00:00:00'))\n",
    "\n",
    "    return ds.get('streamflow').to_pandas().rename('streamflow (m^3 s^-1)')\n",
    "\n",
    "def resample_to_daily(df):\n",
    "    return df.resample('1D').mean()\n",
    "\n",
    "def visualize_summary_statistics(df):\n",
    "    print(df.describe())\n",
    "\n",
    "# Acquire / Filter\n",
    "#df = acquire_streamflow_nwis_iv(site='04294000', start='2022-06-01', end='2022-11-01')\n",
    "df = acquire_streamflow_nwm_retrospective(166176984, '2016-06-01', '2016-11-01')\n",
    "\n",
    "# Manipulate\n",
    "daily = resample_to_daily(df)\n",
    "\n",
    "# Visualize\n",
    "visualize_summary_statistics(daily)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "38794047",
   "metadata": {},
   "source": [
    "## Process Notes\n",
    "I want to take a second to point out how we used Jupyter here to explore the NWM Retrospective data, and print out our dataset as we filtered it along the way to make sure we were doing what we intended. This iteration and exploration is an essential pieces of the software development and data science methodologies. You need to make sure along the way that the data you are getting is the data you think you are getting. Then, once you are confident in your methodology, you can encapsulate that work into a function and parameterize it for future use."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "04e474c2-070c-4971-b83d-d8d4c7664b3c",
   "metadata": {},
   "source": [
    "## Other Potential Data Acquisition Functions\n",
    "Initially, I was going to go over a number of additional data acquisition modules for datasets like AORC, GFS, CFS, and other 2D datasets and forcing data. But, then I decided to focus more on workflow structure and best practices for workflow construction. Plus, as the Python ecosystem continues to develop, we are getting more and more of these data acquisition modules written for us, like the USGS dataretrieval Python module. A number of additional resources for data acquisition are provided in the Wrap Up."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}