0.9.1-beta release (neuralhydrology#4)

* Create readthedocs.yml File is required so that readthedocs can compile the documentation * Fixed external links * fix optional input argument * add github workflows (neuralhydrology#2) * add github workflows * Update .github/workflows/pytest-ci.yml Co-authored-by: Martin Gauch <[email protected]> Co-authored-by: Martin Gauch <[email protected]> * automatic datetime coord inference (neuralhydrology#1) * automatic datetime coord inference * Update neuralhydrology/data/utils.py Co-authored-by: Martin Gauch <[email protected]> Co-authored-by: Martin Gauch <[email protected]> * Added introduction notebook to docs, some docs updates (neuralhydrology#3) * Added introduction notebook * added nbsphinx resources * Apply suggestions from code review Co-authored-by: Martin Gauch <[email protected]> * Update package name, integrated code review comments * Update docs/source/usage/models.rst Co-authored-by: Martin Gauch <[email protected]> Co-authored-by: Martin Gauch <[email protected]> * increment version number Co-authored-by: Martin Gauch <[email protected]>
NASA-LIS · Oct 5, 2020 · 7ae88c5 · 7ae88c5
1 parent 06d1307
commit 7ae88c5
Show file tree

Hide file tree

Showing 22 changed files with 1,309 additions and 112 deletions.
diff --git a/.github/workflows/pytest-ci.yml b/.github/workflows/pytest-ci.yml
@@ -0,0 +1,46 @@
+# Workflow to run the pytest test suite.
+
+name: pytest CI
+
+on:
+  pull_request:
+    branches:
+      - master
+  push:
+    branches:
+      - master
+
+jobs:
+  test:
+    runs-on: ubuntu-latest
+
+    steps:
+      # Checkout the repository under $GITHUB_WORKSPACE
+      - uses: actions/checkout@v2
+
+      # initialize conda
+      - name: Conda setup
+        uses: s-weigand/setup-conda@v1
+        with:
+          update-conda: true
+          python-version: 3.7
+
+      # cache the conda installation to speedup the CI runs
+      - uses: actions/cache@v2
+        id: cache
+        with:
+          path: /usr/share/miniconda/envs/neuralhydrology
+          key: ${{ runner.os }}-conda-cache-${{ hashFiles('environments/environment_cuda10_2.yml') }}
+
+      # on cache miss, create the env from scratch
+      - name: Conda environment creation
+        if: steps.cache.outputs.cache-hit != 'true'
+        run: |
+          conda env create -f environments/environment_cuda10_2.yml
+          source activate neuralhydrology
+
+      # Run the tests
+      - name: Testing with pytest
+        run: |
+          source activate neuralhydrology
+          pytest --cov=neuralhydrology
diff --git a/.gitignore b/.gitignore
@@ -12,8 +12,8 @@ dist/*
 neuralhydrology.egg-info/*
 .vscode/*
 .idea/*
-runs/*
+runs/
 configs/*
-.ipynb_checkpoints/*
+.ipynb_checkpoints/
 data/*
 docs/build/*
diff --git a/README.md b/README.md
@@ -14,8 +14,8 @@ this code in our day-to-day research and will continue to integrate our new rese
 
 **Note:** We will gradually add more examples/documentation over the next couple of days/weeks.
 
-- Documentation: [neuralhydrology.readthedocs.io](neuralhydrology.readthedocs.io)
-- Research Blog: [neuralhydrology.github.io](neuralhydrology.github.io)
+- Documentation: [neuralhydrology.readthedocs.io](https://neuralhydrology.readthedocs.io)
+- Research Blog: [neuralhydrology.github.io](https://neuralhydrology.github.io)
 - Bug reports/Feature requests [https://github.com/neuralhydrology/neuralhydrology/issues](https://github.com/neuralhydrology/neuralhydrology/issues)
 
 # Getting started
@@ -115,4 +115,4 @@ use the ``nh-results-ensemble`` script::
 # Contact
 
 If you have any questions regarding the usage of this repository, feature requests or comments, please open an issue.
-You can also reach out to Frederik Kratzert (kratzert(at)ml.jku.at) by email.
+You can also reach out to Frederik Kratzert (kratzert(at)ml.jku.at) by email.
diff --git a/docs/source/api/neuralhydrology.rst b/docs/source/api/neuralhydrology.rst
@@ -1,5 +1,5 @@
-neuralhydrology
-===============
+neuralhydrology API
+===================
 
 .. automodule:: neuralhydrology
    :members:

diff --git a/docs/source/conf.py b/docs/source/conf.py
@@ -10,18 +10,22 @@
 # add these directories to sys.path here. If the directory is relative to the
 # documentation root, use os.path.abspath to make it absolute, like shown here.
 #
+import datetime
 import os
 import sys
 sys.path.insert(0, os.path.abspath('.'))
 sys.path.insert(0, os.path.abspath('../../'))
 # -- Project information -----------------------------------------------------
+about = {}
+with open('../../neuralhydrology/__about__.py', "r") as fp:
+    exec(fp.read(), about)
 
-project = 'NeuralHydrology'
-copyright = '2020, Frederik Kratzert'
+project = 'neuralHydrology'
+copyright = f'{datetime.datetime.now().year}, Frederik Kratzert'
 author = 'Frederik Kratzert'
 
 # The full version, including alpha/beta/rc tags
-release = '0.9.0-beta'
+release = about["__version__"]
 
 # -- General configuration ---------------------------------------------------
 
@@ -31,7 +35,9 @@
 extensions = [
     'sphinx.ext.autodoc',  # autodocument
     'sphinx.ext.napoleon',  # google and numpy doc string support
-    'sphinx.ext.mathjax'  # latex rendering of equations using MathJax
+    'sphinx.ext.mathjax',  # latex rendering of equations using MathJax
+    'nbsphinx',  # for direct embedding of jupyter notebooks into sphinx docs
+    'nbsphinx_link'  # to be able to include notebooks from outside of the docs folder
 ]
 
 # Add any paths that contain templates here, relative to this directory.
@@ -40,7 +46,7 @@
 # List of patterns, relative to source directory, that match files and
 # directories to ignore when looking for source files.
 # This pattern also affects html_static_path and html_extra_path.
-exclude_patterns = []
+exclude_patterns = ['**.ipynb_checkpoints']
 
 # -- Options for HTML output -------------------------------------------------
 

diff --git a/docs/source/index.rst b/docs/source/index.rst
@@ -3,13 +3,17 @@
    You can adapt this file completely to your liking, but it should at least
    contain the root `toctree` directive.
 
-Welcome to NeuralHydrology's documentation!
+Welcome to neuralHydrology's documentation!
 ===========================================
 
+The documentation is still work-in-progress. Stay tuned for a lot of updates during the next days/weeks, as well as
+a handful of tutorials.
+
 .. toctree::
-   :maxdepth: 5
+   :maxdepth: 2
    :caption: Contents:
 
    usage/quickstart
    usage/models
+   tutorials/index
    api/neuralhydrology
diff --git a/docs/source/tutorials/index.rst b/docs/source/tutorials/index.rst
@@ -0,0 +1,11 @@
+Tutorials
+---------
+
+We will gradually add more tutorials over the next couple of weeks, highlighting some of
+the functionality of this Python package.
+
+.. toctree::
+   :maxdepth: 1
+   :caption: Contents:
+
+   introduction
diff --git a/docs/source/tutorials/introduction.nblink b/docs/source/tutorials/introduction.nblink
@@ -0,0 +1,3 @@
+{
+    "path": "../../../examples/01-Introduction/Introduction.ipynb"
+}
diff --git a/docs/source/usage/models.rst b/docs/source/usage/models.rst
@@ -1,28 +1,29 @@
-Models
-======
+Modelzoo
+========
 
-The following section gives an overview over all implemented models.
+The following section gives an overview of all implemented models. See `Implementing a new model`_ for details
+on how to add your own model to the neuralHydrology package.
+
+BaseModel
+---------
+Abstract base class from which all models derive. Do not use this class for model training.
 
 CudaLSTM
 --------
 :py:class:`neuralhydrology.modelzoo.cudalstm.CudaLSTM` is a network using the standard PyTorch LSTM implementation.
-All features (``x_d``, ``x_s``, ``x_one_hot``) are concatenated and passed at each time step.
-Initial forget gate bias can be set (in config.yml) and will be set during model initialization.
+All features (``x_d``, ``x_s``, ``x_one_hot``) are concatenated and passed to the network at each time step.
+The initial forget gate bias can be defined in config.yml (``initial_forget_bias``) and will be set accordingly during
+model initialization.
 
 EA-LSTM
 -------
-:py:class:`neuralhydrology.modelzoo.ealstm.EALSTM` is an implementation of the Entity-Aware LSTM, as used
-in the 2019 HESS paper. The static features (``x_s`` and/or ``x_one_hot``) are used to compute the input gate
-activations, while ``x_d`` is used in all other gates of the network.
-Initial forget gate bias can be set, and if ``embedding_hiddens`` is passed, the input gate consists of the so-defined
+:py:class:`neuralhydrology.modelzoo.ealstm.EALSTM` is an implementation of the Entity-Aware LSTM, as introduced in
+`Kratzert et al. "Towards learning universal, regional, and local hydrological behaviors via machine learning applied to large-sample datasets" <https://hess.copernicus.org/articles/23/5089/2019/hess-23-5089-2019.html>`__.
+The static features (``x_s`` and/or ``x_one_hot``) are used to compute the input gate activations, while the dynamic
+inputs ``x_d`` are used in all other gates of the network.
+The initial forget gate bias can be defined in config.yml (``initial_forget_bias``). If ``embedding_hiddens`` is passed, the input gate consists of the so-defined
 FC network and not a single linear layer.
 
-LSTM
-----
-:py:class:`neuralhydrology.modelzoo.lstm.LSTM` is an own LSTM implementation.
-Momentarily, the only advantage compared to CudaLSTM is the return of the entire cell state array.
-This class will be most likely adapted/changed in the near future to provide much more flexibility for various settings.
-
 EmbCudaLSTM
 -----------
 :py:class:`neuralhydrology.modelzoo.embcudalstm.EmbCudaLSTM` is similar to `CudaLSTM`_,
@@ -31,14 +32,34 @@ with the only difference that static inputs (``x_s`` and/or ``x_one_hot``) are p
 at each time step.
 
 
+LSTM
+----
+:py:class:`neuralhydrology.modelzoo.lstm.LSTM` is a PyTorch port of the CudaLSTM that returns all gate and state
+activations for all time steps. This class is implemented for exploratory reasons. You can use the method
+``model.copy_weights()`` to copy the weights of a ``CudaLSTM`` model into an ``LSTM`` model. This allows to use the fast
+CUDA implementation for training, and only use this class for inference with more detailed outputs.
+
+MultiFreqLSTM
+-------------
+:py:class:`neuralhydrology.modelzoo.multifreqlstm.MultiFreqLSTM` is a newly proposed model by Gauch et al. (pre-print
+published soon). This model allows the training on more than one temporal frequency (e.g. daily and hourly inputs) and
+returns multi-frequency model predictions accordingly. A more detailed tutorial will follow shortly.
+
+ODELSTM
+-------
+:py:class:`neuralhydrology.modelzoo.odelstm.ODELSTM` is a PyTorch implementation of the ODE-LSTM proposed by
+`Lechner and Hasani <https://arxiv.org/abs/2006.04418>`_. This model can be used with unevenly sampled inputs and can
+be queried to return predictions for any arbitrary time step.
+
+
 Implementing a new model
 ------------------------
 The listing below shows the skeleton of a template model you can use to start implementing your own model.
 Once you have implemented your model, make sure to modify :py:func:`neuralhydrology.modelzoo.__init__.get_model`.
 Furthermore, make sure to select a *unique* model abbreviation that will be used to specify the model in the config.yml
 files.
 
-::
+.. code-block:: python
 
     from typing import Dict
 
@@ -105,4 +126,3 @@ files.
             # Implement forward pass here #
             ###############################
             pass
-
diff --git a/docs/source/usage/quickstart.rst b/docs/source/usage/quickstart.rst
@@ -3,18 +3,17 @@ Quick Start
 
 Installation
 ------------
-The neuralhydrology project is available on PyPI.
-Hence, installation is as easy as::
+For now, download or clone the repository to your local machine and install a local, editable copy.
+This is a good idea if you want to edit the ``neuralhydrology`` code (e.g., adding new models or datasets)
 
-    pip install neuralhydrology
+.. code-block::
 
-Alternatively, you can clone the repository and install the local, editable copy. This is a good idea if you want to
-edit the ``neuralhydrology`` code (e.g., adding new models or datasets).::
-
-    git clone https://github.com/kratzert/lstm_based_hydrology.git
-    cd lstm_based_hydrology
+    git clone https://github.com/neuralhydrology/neuralhydrology.git
+    cd neuralhydrology
     pip install -e .
 
+Besides adding the package to your Python environment, it will also add three bash scripts:
+`nh-run`, `nh-run-scheduler` and `nh-results-ensemble`. For details, see below.
 
 Data
 ----
@@ -35,7 +34,7 @@ To train a model, prepare a configuration file, then run::
 If you want to train multiple models, you can make use of the ``nh-run-scheduler`` command.
 Place all configs in a folder, then run::
 
-    nh-run-scheduler --config-dir /path/to/config_dir/ --runs-per-gpu X --gpu-ids Y
+    nh-run-scheduler train --config-dir /path/to/config_dir/ --runs-per-gpu X --gpu-ids Y
 
 With X, you can specify how many models should be trained on parallel on a single GPU.
 With Y, you can specify which GPUs to use for training (use the id as specified in ``nvidia-smi``).
@@ -52,7 +51,7 @@ the weights of the last epoch are used.
 
 To evaluate all runs in a specific directory you can, similarly to training, run::
 
-    nh-run-scheduler --mode evaluate --run-dir /path/to/config_dir/ --runs-per-gpu X --gpu-ids Y
+    nh-run-scheduler evaluate --run-dir /path/to/config_dir/ --runs-per-gpu X --gpu-ids Y
 
 
 To merge the predictons of a number of runs (stored in ``$DIR1``, ...) into one averaged ensemble,

diff --git a/environments/environment_cpu.yml b/environments/environment_cpu.yml
@@ -27,4 +27,5 @@ dependencies:
   - pip:
     - tensorboard
     - sphinx-rtd-theme
-
+    - nbsphinx
+    - nbsphinx-link
diff --git a/environments/environment_cuda10_2.yml b/environments/environment_cuda10_2.yml
@@ -27,4 +27,5 @@ dependencies:
   - pip:
     - tensorboard
     - sphinx-rtd-theme
-
+    - nbsphinx
+    - nbsphinx-link
diff --git a/environments/environment_cuda9_2.yml b/environments/environment_cuda9_2.yml
@@ -27,4 +27,5 @@ dependencies:
   - pip:
     - tensorboard
     - sphinx-rtd-theme
-
+    - nbsphinx
+    - nbsphinx-link
diff --git a/examples/01-Introduction/1_basin.txt b/examples/01-Introduction/1_basin.txt
@@ -0,0 +1 @@
+01022500