Skip to content

Commit

Permalink
0.9.1-beta release (neuralhydrology#1)
Browse files Browse the repository at this point in the history
  • Loading branch information
kratzert authored Oct 5, 2020
2 parents 00b81bd + 7ae88c5 commit 14c27c9
Show file tree
Hide file tree
Showing 20 changed files with 1,304 additions and 109 deletions.
46 changes: 46 additions & 0 deletions .github/workflows/pytest-ci.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,46 @@
# Workflow to run the pytest test suite.

name: pytest CI

on:
pull_request:
branches:
- master
push:
branches:
- master

jobs:
test:
runs-on: ubuntu-latest

steps:
# Checkout the repository under $GITHUB_WORKSPACE
- uses: actions/checkout@v2

# initialize conda
- name: Conda setup
uses: s-weigand/setup-conda@v1
with:
update-conda: true
python-version: 3.7

# cache the conda installation to speedup the CI runs
- uses: actions/cache@v2
id: cache
with:
path: /usr/share/miniconda/envs/neuralhydrology
key: ${{ runner.os }}-conda-cache-${{ hashFiles('environments/environment_cuda10_2.yml') }}

# on cache miss, create the env from scratch
- name: Conda environment creation
if: steps.cache.outputs.cache-hit != 'true'
run: |
conda env create -f environments/environment_cuda10_2.yml
source activate neuralhydrology
# Run the tests
- name: Testing with pytest
run: |
source activate neuralhydrology
pytest --cov=neuralhydrology
4 changes: 2 additions & 2 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -12,8 +12,8 @@ dist/*
neuralhydrology.egg-info/*
.vscode/*
.idea/*
runs/*
runs/
configs/*
.ipynb_checkpoints/*
.ipynb_checkpoints/
data/*
docs/build/*
4 changes: 2 additions & 2 deletions docs/source/api/neuralhydrology.rst
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
neuralhydrology
===============
neuralhydrology API
===================

.. automodule:: neuralhydrology
:members:
Expand Down
16 changes: 11 additions & 5 deletions docs/source/conf.py
Original file line number Diff line number Diff line change
Expand Up @@ -10,18 +10,22 @@
# add these directories to sys.path here. If the directory is relative to the
# documentation root, use os.path.abspath to make it absolute, like shown here.
#
import datetime
import os
import sys
sys.path.insert(0, os.path.abspath('.'))
sys.path.insert(0, os.path.abspath('../../'))
# -- Project information -----------------------------------------------------
about = {}
with open('../../neuralhydrology/__about__.py', "r") as fp:
exec(fp.read(), about)

project = 'NeuralHydrology'
copyright = '2020, Frederik Kratzert'
project = 'neuralHydrology'
copyright = f'{datetime.datetime.now().year}, Frederik Kratzert'
author = 'Frederik Kratzert'

# The full version, including alpha/beta/rc tags
release = '0.9.0-beta'
release = about["__version__"]

# -- General configuration ---------------------------------------------------

Expand All @@ -31,7 +35,9 @@
extensions = [
'sphinx.ext.autodoc', # autodocument
'sphinx.ext.napoleon', # google and numpy doc string support
'sphinx.ext.mathjax' # latex rendering of equations using MathJax
'sphinx.ext.mathjax', # latex rendering of equations using MathJax
'nbsphinx', # for direct embedding of jupyter notebooks into sphinx docs
'nbsphinx_link' # to be able to include notebooks from outside of the docs folder
]

# Add any paths that contain templates here, relative to this directory.
Expand All @@ -40,7 +46,7 @@
# List of patterns, relative to source directory, that match files and
# directories to ignore when looking for source files.
# This pattern also affects html_static_path and html_extra_path.
exclude_patterns = []
exclude_patterns = ['**.ipynb_checkpoints']

# -- Options for HTML output -------------------------------------------------

Expand Down
8 changes: 6 additions & 2 deletions docs/source/index.rst
Original file line number Diff line number Diff line change
Expand Up @@ -3,13 +3,17 @@
You can adapt this file completely to your liking, but it should at least
contain the root `toctree` directive.
Welcome to NeuralHydrology's documentation!
Welcome to neuralHydrology's documentation!
===========================================

The documentation is still work-in-progress. Stay tuned for a lot of updates during the next days/weeks, as well as
a handful of tutorials.

.. toctree::
:maxdepth: 5
:maxdepth: 2
:caption: Contents:

usage/quickstart
usage/models
tutorials/index
api/neuralhydrology
11 changes: 11 additions & 0 deletions docs/source/tutorials/index.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
Tutorials
---------

We will gradually add more tutorials over the next couple of weeks, highlighting some of
the functionality of this Python package.

.. toctree::
:maxdepth: 1
:caption: Contents:

introduction
3 changes: 3 additions & 0 deletions docs/source/tutorials/introduction.nblink
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
{
"path": "../../../examples/01-Introduction/Introduction.ipynb"
}
54 changes: 37 additions & 17 deletions docs/source/usage/models.rst
Original file line number Diff line number Diff line change
@@ -1,28 +1,29 @@
Models
======
Modelzoo
========

The following section gives an overview over all implemented models.
The following section gives an overview of all implemented models. See `Implementing a new model`_ for details
on how to add your own model to the neuralHydrology package.

BaseModel
---------
Abstract base class from which all models derive. Do not use this class for model training.

CudaLSTM
--------
:py:class:`neuralhydrology.modelzoo.cudalstm.CudaLSTM` is a network using the standard PyTorch LSTM implementation.
All features (``x_d``, ``x_s``, ``x_one_hot``) are concatenated and passed at each time step.
Initial forget gate bias can be set (in config.yml) and will be set during model initialization.
All features (``x_d``, ``x_s``, ``x_one_hot``) are concatenated and passed to the network at each time step.
The initial forget gate bias can be defined in config.yml (``initial_forget_bias``) and will be set accordingly during
model initialization.

EA-LSTM
-------
:py:class:`neuralhydrology.modelzoo.ealstm.EALSTM` is an implementation of the Entity-Aware LSTM, as used
in the 2019 HESS paper. The static features (``x_s`` and/or ``x_one_hot``) are used to compute the input gate
activations, while ``x_d`` is used in all other gates of the network.
Initial forget gate bias can be set, and if ``embedding_hiddens`` is passed, the input gate consists of the so-defined
:py:class:`neuralhydrology.modelzoo.ealstm.EALSTM` is an implementation of the Entity-Aware LSTM, as introduced in
`Kratzert et al. "Towards learning universal, regional, and local hydrological behaviors via machine learning applied to large-sample datasets" <https://hess.copernicus.org/articles/23/5089/2019/hess-23-5089-2019.html>`__.
The static features (``x_s`` and/or ``x_one_hot``) are used to compute the input gate activations, while the dynamic
inputs ``x_d`` are used in all other gates of the network.
The initial forget gate bias can be defined in config.yml (``initial_forget_bias``). If ``embedding_hiddens`` is passed, the input gate consists of the so-defined
FC network and not a single linear layer.

LSTM
----
:py:class:`neuralhydrology.modelzoo.lstm.LSTM` is an own LSTM implementation.
Momentarily, the only advantage compared to CudaLSTM is the return of the entire cell state array.
This class will be most likely adapted/changed in the near future to provide much more flexibility for various settings.

EmbCudaLSTM
-----------
:py:class:`neuralhydrology.modelzoo.embcudalstm.EmbCudaLSTM` is similar to `CudaLSTM`_,
Expand All @@ -31,14 +32,34 @@ with the only difference that static inputs (``x_s`` and/or ``x_one_hot``) are p
at each time step.


LSTM
----
:py:class:`neuralhydrology.modelzoo.lstm.LSTM` is a PyTorch port of the CudaLSTM that returns all gate and state
activations for all time steps. This class is implemented for exploratory reasons. You can use the method
``model.copy_weights()`` to copy the weights of a ``CudaLSTM`` model into an ``LSTM`` model. This allows to use the fast
CUDA implementation for training, and only use this class for inference with more detailed outputs.

MultiFreqLSTM
-------------
:py:class:`neuralhydrology.modelzoo.multifreqlstm.MultiFreqLSTM` is a newly proposed model by Gauch et al. (pre-print
published soon). This model allows the training on more than one temporal frequency (e.g. daily and hourly inputs) and
returns multi-frequency model predictions accordingly. A more detailed tutorial will follow shortly.

ODELSTM
-------
:py:class:`neuralhydrology.modelzoo.odelstm.ODELSTM` is a PyTorch implementation of the ODE-LSTM proposed by
`Lechner and Hasani <https://arxiv.org/abs/2006.04418>`_. This model can be used with unevenly sampled inputs and can
be queried to return predictions for any arbitrary time step.


Implementing a new model
------------------------
The listing below shows the skeleton of a template model you can use to start implementing your own model.
Once you have implemented your model, make sure to modify :py:func:`neuralhydrology.modelzoo.__init__.get_model`.
Furthermore, make sure to select a *unique* model abbreviation that will be used to specify the model in the config.yml
files.

::
.. code-block:: python
from typing import Dict
Expand Down Expand Up @@ -105,4 +126,3 @@ files.
# Implement forward pass here #
###############################
pass

19 changes: 9 additions & 10 deletions docs/source/usage/quickstart.rst
Original file line number Diff line number Diff line change
Expand Up @@ -3,18 +3,17 @@ Quick Start

Installation
------------
The neuralhydrology project is available on PyPI.
Hence, installation is as easy as::
For now, download or clone the repository to your local machine and install a local, editable copy.
This is a good idea if you want to edit the ``neuralhydrology`` code (e.g., adding new models or datasets)

pip install neuralhydrology
.. code-block::
Alternatively, you can clone the repository and install the local, editable copy. This is a good idea if you want to
edit the ``neuralhydrology`` code (e.g., adding new models or datasets).::

git clone https://github.com/kratzert/lstm_based_hydrology.git
cd lstm_based_hydrology
git clone https://github.com/neuralhydrology/neuralhydrology.git
cd neuralhydrology
pip install -e .
Besides adding the package to your Python environment, it will also add three bash scripts:
`nh-run`, `nh-run-scheduler` and `nh-results-ensemble`. For details, see below.

Data
----
Expand All @@ -35,7 +34,7 @@ To train a model, prepare a configuration file, then run::
If you want to train multiple models, you can make use of the ``nh-run-scheduler`` command.
Place all configs in a folder, then run::

nh-run-scheduler --config-dir /path/to/config_dir/ --runs-per-gpu X --gpu-ids Y
nh-run-scheduler train --config-dir /path/to/config_dir/ --runs-per-gpu X --gpu-ids Y

With X, you can specify how many models should be trained on parallel on a single GPU.
With Y, you can specify which GPUs to use for training (use the id as specified in ``nvidia-smi``).
Expand All @@ -52,7 +51,7 @@ the weights of the last epoch are used.

To evaluate all runs in a specific directory you can, similarly to training, run::

nh-run-scheduler --mode evaluate --run-dir /path/to/config_dir/ --runs-per-gpu X --gpu-ids Y
nh-run-scheduler evaluate --run-dir /path/to/config_dir/ --runs-per-gpu X --gpu-ids Y


To merge the predictons of a number of runs (stored in ``$DIR1``, ...) into one averaged ensemble,
Expand Down
3 changes: 2 additions & 1 deletion environments/environment_cpu.yml
Original file line number Diff line number Diff line change
Expand Up @@ -27,4 +27,5 @@ dependencies:
- pip:
- tensorboard
- sphinx-rtd-theme

- nbsphinx
- nbsphinx-link
3 changes: 2 additions & 1 deletion environments/environment_cuda10_2.yml
Original file line number Diff line number Diff line change
Expand Up @@ -27,4 +27,5 @@ dependencies:
- pip:
- tensorboard
- sphinx-rtd-theme

- nbsphinx
- nbsphinx-link
3 changes: 2 additions & 1 deletion environments/environment_cuda9_2.yml
Original file line number Diff line number Diff line change
Expand Up @@ -27,4 +27,5 @@ dependencies:
- pip:
- tensorboard
- sphinx-rtd-theme

- nbsphinx
- nbsphinx-link
1 change: 1 addition & 0 deletions examples/01-Introduction/1_basin.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
01022500
Loading

0 comments on commit 14c27c9

Please sign in to comment.