Updates RSL-RL to version 2.0 (#14)

* Adds files from internal
* Updates version to 2.0
* Fix to split and pad for rnn policies (#15)
* Small fix to adapt to new configs from orbit (#16)
* Fixes dummy config
* Adds onnx to deps
* Adds contributors

---------

Co-authored-by: Nikita Rudin <[email protected]>

.flake8

@@ -0,0 +1,22 @@
+[flake8]
+show-source=True
+statistics=True
+per-file-ignores=*/__init__.py:F401
+# E402: Module level import not at top of file
+# E501: Line too long
+# W503: Line break before binary operator
+# E203: Whitespace before ':' -> conflicts with black
+# D401: First line should be in imperative mood
+# R504: Unnecessary variable assignment before return statement.
+# R505: Unnecessary elif after return statement
+# SIM102: Use a single if-statement instead of nested if-statements
+# SIM117: Merge with statements for context managers that have same scope.
+ignore=E402,E501,W503,E203,D401,R504,R505,SIM102,SIM117
+max-line-length = 120
+max-complexity = 18
+exclude=_*,.vscode,.git,docs/**
+# docstrings
+docstring-convention=google
+# annotations
+suppress-none-returning=True
+allow-star-arg-any=True

.gitignore

@@ -9,4 +9,4 @@ __pycache__
 .pytest_cache
 
 # vs code
-.vscode
+.vscode

.pre-commit-config.yaml

@@ -0,0 +1,58 @@
+repos:
+  - repo: https://github.com/python/black
+    rev: 23.10.1
+    hooks:
+      - id: black
+        args: ["--line-length", "120", "--preview"]
+  - repo: https://github.com/pycqa/flake8
+    rev: 6.1.0
+    hooks:
+      - id: flake8
+        additional_dependencies: [flake8-simplify, flake8-return]
+  - repo: https://github.com/pre-commit/pre-commit-hooks
+    rev: v4.5.0
+    hooks:
+      - id: trailing-whitespace
+      - id: check-symlinks
+      - id: destroyed-symlinks
+      - id: check-yaml
+      - id: check-merge-conflict
+      - id: check-case-conflict
+      - id: check-executables-have-shebangs
+      - id: check-toml
+      - id: end-of-file-fixer
+      - id: check-shebang-scripts-are-executable
+      - id: detect-private-key
+      - id: debug-statements
+  - repo: https://github.com/pycqa/isort
+    rev: 5.12.0
+    hooks:
+      - id: isort
+        name: isort (python)
+        args: ["--profile", "black", "--filter-files"]
+  - repo: https://github.com/asottile/pyupgrade
+    rev: v3.15.0
+    hooks:
+      - id: pyupgrade
+        args: ["--py37-plus"]
+  - repo: https://github.com/codespell-project/codespell
+    rev: v2.2.6
+    hooks:
+      - id: codespell
+        additional_dependencies:
+        - tomli
+  # FIXME: Figure out why this is getting stuck under VPN.
+  # - repo: https://github.com/RobertCraigie/pyright-python
+  #   rev: v1.1.315
+  #   hooks:
+  #   - id: pyright
+  # Note: We disable this by default since not all code is compatible with it.
+  # - repo: https://github.com/Lucas-C/pre-commit-hooks
+  #   rev: v1.5.1
+  #   hooks:
+  #     - id: insert-license
+  #       files: \.py$
+  #       args:
+  #         # - --remove-header    # Remove existing license headers. Useful when updating license.
+  #         - --license-filepath
+  #         - .github/LICENSE_HEADER.txt

CONTRIBUTORS.md

@@ -0,0 +1,31 @@
+# RSL-RL Maintainers and Contributors
+
+This is the official list of developers and contributors.
+
+To see the full list of contributors, see the revision history in the source control.
+
+Names should be added to this file as: individual names or organizations.
+
+Email addresses are tracked elsewhere to avoid spam.
+
+Please keep the lists sorted alphabetically.
+
+## Maintainers
+
+* Robotic Syetms Lab, ETH Zurich
+* NVIDIA Corporation
+
+---
+
+* David Hoeller
+* Nikita Rudin
+
+## Contributors
+
+* Eric Vollenweider
+* Fabian Jenelten
+* Lorenzo Terenzi
+* Marko Bjelonic
+* Matthijs van der Boon
+* Mayank Mittal
+* Zhang Chong

LICENSE

@@ -27,4 +27,4 @@ ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
 (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS
 SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
 
-See licenses/dependencies for license information of dependencies of this package.
+See licenses/dependencies for license information of dependencies of this package.

README.md

@@ -1,26 +1,57 @@
 # RSL RL
+
 Fast and simple implementation of RL algorithms, designed to run fully on GPU.
 This code is an evolution of `rl-pytorch` provided with NVIDIA's Isaac GYM.
 
 Only PPO is implemented for now. More algorithms will be added later.
 Contributions are welcome.
 
+**Maintainer**: David Hoeller and Nikita Rudin <br/>
+**Affiliation**: Robotic Systems Lab, ETH Zurich & NVIDIA <br/>
+**Contact**: [email protected]
+
 ## Setup
 
-```
+Following are the instructions to setup the repository for your workspace:
+
+```bash
 git clone https://github.com/leggedrobotics/rsl_rl
 cd rsl_rl
 pip install -e .
 ```
 
-### Useful Links ###
-Example use case: https://github.com/leggedrobotics/legged_gym  
-Project website: https://leggedrobotics.github.io/legged_gym/  
-Paper: https://arxiv.org/abs/2109.11978
+The framework supports the following logging frameworks which can be configured through `logger`:
+
+* Tensorboard: https://www.tensorflow.org/tensorboard/
+* Weights & Biases: https://wandb.ai/site
+* Neptune: https://docs.neptune.ai/
+
+For a demo configuration of the PPO, please check: [dummy_config.yaml](config/dummy_config.yaml) file.
+
 
-**Maintainer**: Nikita Rudin  
-**Affiliation**: Robotic Systems Lab, ETH Zurich & NVIDIA  
-**Contact**: [email protected]  
+## Contribution Guidelines
+
+For documentation, we adopt the [Google Style Guide](https://sphinxcontrib-napoleon.readthedocs.io/en/latest/example_google.html) for docstrings. We use [Sphinx](https://www.sphinx-doc.org/en/master/) for generating the documentation. Please make sure that your code is well-documented and follows the guidelines.
+
+We use the following tools for maintaining code quality:
+
+- [pre-commit](https://pre-commit.com/): Runs a list of formatters and linters over the codebase.
+- [black](https://black.readthedocs.io/en/stable/): The uncompromising code formatter.
+- [flake8](https://flake8.pycqa.org/en/latest/): A wrapper around PyFlakes, pycodestyle, and McCabe complexity checker.
+
+Please check [here](https://pre-commit.com/#install) for instructions to set these up. To run over the entire repository, please execute the following command in the terminal:
+
+
+```bash
+# for installation (only once)
+pre-commit install
+# for running
+pre-commit run --all-files
+```
 
+### Useful Links
 
+Environment repositories using the framework:
 
+* `Legged-Gym` (built on top of NVIDIA Isaac Gym): https://leggedrobotics.github.io/legged_gym/
+* `Orbit` (built on top of NVIDIA Isaac Sim): https://isaac-orbit.github.io/

config/dummy_config.yaml

@@ -0,0 +1,48 @@
+algorithm:
+  class_name: PPO
+  # training parameters
+  # -- value function
+  value_loss_coef: 1.0
+  clip_param: 0.2
+  use_clipped_value_loss: true
+  # -- surrogate loss
+  desired_kl: 0.01
+  entropy_coef: 0.01
+  gamma: 0.99
+  lam: 0.95
+  max_grad_norm: 1.0
+  # -- training
+  learning_rate: 0.001
+  num_learning_epochs: 5
+  num_mini_batches: 4  # mini batch size = num_envs * num_steps / num_mini_batches
+  schedule: adaptive  # adaptive, fixed
+policy:
+  class_name: ActorCritic
+  # for MLP i.e. `ActorCritic`
+  activation: elu
+  actor_hidden_dims: [128, 128, 128]
+  critic_hidden_dims: [128, 128, 128]
+  init_noise_std: 1.0
+  # only needed for `ActorCriticRecurrent`
+  # rnn_type: 'lstm'
+  # rnn_hidden_size: 512
+  # rnn_num_layers: 1
+runner:
+    num_steps_per_env: 24  # number of steps per environment per iteration
+    max_iterations: 1500  # number of policy updates
+    empirical_normalization: false
+    # -- logging parameters
+    save_interval: 50  # check for potential saves every `save_interval` iterations
+    experiment_name: walking_experiment
+    run_name: ""
+    # -- logging writer
+    logger: tensorboard  # tensorboard, neptune, wandb
+    neptune_project: legged_gym
+    wandb_project: legged_gym
+    # -- load and resuming
+    resume: false
+    load_run: -1  # -1 means load latest run
+    resume_path: null  # updated from load_run and checkpoint
+    checkpoint: -1  # -1 means load latest checkpoint
+runner_class_name: OnPolicyRunner
+seed: 1

licenses/dependencies/black-license.txt

@@ -0,0 +1,21 @@
+The MIT License (MIT)
+
+Copyright (c) 2018 Łukasz Langa
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.