Skip to content

Commit

Permalink
Added DiskStorage class to allow storing checkpoins into disk.
Browse files Browse the repository at this point in the history
    Wrapper around libadjoint Revolve implementations. Now contains online and multistage checkpointing. C++ code needs cleanup to remove warnings and missing functions.

    updated readme to reflect new interface

    removed function that is missing in C implementation from wrapper

    removed sections that only applied to the old wrapper

    improved wrapper: error handling, moved helper functions outside, as they are supposed to help the user find the right sizes /before/ creating an object

    added a convenience function that takes operators and handles the checkpointing

    functioning prototype for the devito-like Revolve wrapper

    cleanup in the wrapper, removed some print statements

    updated license, prettified code.

    nicer examples to illustrate Revolve

    slightly improved example code formatting

    flake8 is happy

    added check if symbols are passed as a dictionary, made Flake8 happy, made CRevolve a separete instance for each pyrevolve object.

    Rewrote API to improve separation of concerns. Fixes Issue devitocodes#2. We can now pass operators and a checkpoint object, pyrevolve does the rest.

    Update README.md

    updated readme to reflect new API

    Fixed compilation error. This may cause race conditions when more than one CRevolve object is used at the same time.

    add missing file

    removed file again

    Fix for python3

    Add Operator ABC + change to new-style classes

    Add dtype to CheckpointStorage and add extra forward to align computations

    Update example to pass dtype

    Adding a requirements file and some basic travis testing.

    Still trying to get travis working

    Flake

    Hide the crevolve import in setup.py to give setuptools a chance to install Cython if not present

    Setup:  is not a package

    Test python 2 as well as 3

    Import python2 style if python 3 style fails

    suppress uncalled for verbosity

    Fix issue 4: wrong argument order in multistage

    Add versioneer for automatic version numbering

    Add package metadata to setup.py

    Add pyx files to MANIFEST.in

    Make sure the provided action is something we understand

    Add basic testing

    Enable testing

    Flake

    Move utility functions out so they can be reused between tests

    Add blank line

    Ignore errors in versioneer

    Add initial infra for compression

    Separate the scheduler

    add blosc compression

    Add configurable compression params

    Some cleanup

    improving parameter handling

    Add tests for compression

    Allow parameters to be passed to the compression scheme

    Add ZFP compression scheme

    Flake

    Add a complete forward-backward test that can be expanded to test compression

    Add blosc to requirements

    Dirty hack to get travis to use zfp (because I'm too lazy to build a proper installation procedure for pyzfp)

    Fix flake issues

    Add contexttimer to reqs

    The complete forward-backward now works with zfp

    Changing APIs to hide the details of ByteStorage and compression

    All tests passing with new API

    All tests pass

    Add some logging + custom pickling functionality. This version works fine

    Some cleanup

    Add built-in profiler for pyrevolve

    Add counts to profiling

    Add more options to profiler

    Change to using copyto to copy

    Suppress console output from Revolve

    Change the API such that no excess data is copied over

    Add tools

    Profile storage

    Clean up by moving some code into separate files

    Adding missing files

    Some more changes

    Flake fixes

    Compression seems to work

    Enable testing on multiple GCC versions

    Still trying to get travis to accept osx+python combo

    Fix modern example

    Add requirements to Manifest

    Fix to cython version 2

    Fix typo

    Remove pxd file

    Add back flake dependency to continue lint on CI

    Change the reference to tools

    Add static schedule generator

    Better directory structure and README

    Flakey changes

    Make travis test pip installation

    Update tests and travis build

    Update travis build

    Create pythonpublish.yml

    Update README

    Create CI workflow using GitHub Actions

    remove requirements and flake8

    add flake8 & pytest-linux/macOS

    Added DiskStorage class to storage.py to store checkpoints to disk

    Adding DiskStorage class to storage.py file

    Created DiskRevolver and MemoryRevolver classes from Revolver. Renamed Revolver to BaseRevolver. Kept the name Revolver as MemoryRevolver for backward compatibility.

    The following changes were performed:

    1 - Revolver class was split into:
            - BaseRevolver, which implements the the revolver
            FSM and receives a storage object as parameter;
            - DiskRevolver is an specialization of BaseRevolver
            that uses DiskStorage as storage method.
            - MemoryRevolver is an specialization of BaseRevolver
            that uses MemoryStorage as storage method.
            - The name Revolver was kept in the module
            for backward compatibility. If the user uses
            the name Revolver it reverts to MemoryRevolver as original.

    2 - DiskStorage class implements the same
    interface as NumpyStorage, but saving data to
    a .dat file. The file path can be passed as parameter.

    3 - Two test cases were added to validate memory and disk revolver classes.

    Removed bin files from tests/

    Separated R/W file streams in storage.py. Re-wrote test_diskckp.py and test_memckp.py removing devito dependency.

    Fixed some flake8 related format issues

    Fixed some other flake8 related format issues

    Added thread Id to DiskStorage .dat file name; DiskStorage class now receives 'filedir' instead of 'filename'; BaseRevolver class was made abstract, forcing the user to instatiate either Memory or DiskRevolver objects.

    Added 'multi/singlefile' option to DiskRevovler and DiskStorage classes, enabling the user to store checkpoints in a single file or with one file per checkpoint; Added new tests for diskckp using multiple files.

    Using PID to identify checkpoint files uniquely instead of Thread ID

    Using PID to identify checkpoint files uniquely instead of Thread ID

    Removed 'print' calls; Added 'singlefile' parameter to test_diskckp.py to remove redundant tests

    Joined test_memckp.py and test_diskckp.py files using diskckp flag

    Move tests in test_diskckp.py to test_storage.py

    Removed bin files from tests/

    Separated R/W file streams in storage.py. Re-wrote test_diskckp.py and test_memckp.py removing devito dependency.

    Added thread Id to DiskStorage .dat file name; DiskStorage class now receives 'filedir' instead of 'filename'; BaseRevolver class was made abstract, forcing the user to instatiate either Memory or DiskRevolver objects.

    Using PID to identify checkpoint files uniquely instead of Thread ID

    suppress uncalled for verbosity

    Fix issue 4: wrong argument order in multistage

    Add pyx files to MANIFEST.in

    Add blank line

    Separate the scheduler

    add blosc compression

    Add more options to profiler

    Change to using copyto to copy

    Add tools

    Still trying to get travis to accept osx+python combo

    Fix modern example

    Add requirements to Manifest

    Fix to cython version 2

    Fix typo

    Remove pxd file

    Add back flake dependency to continue lint on CI

    Change the reference to tools

    Add static schedule generator

    Better directory structure and README

    Flakey changes

    Update travis build

    Create pythonpublish.yml

    Update README

    Removed bin files from tests/

    Create CI workflow using GitHub Actions

    remove requirements and flake8

    add flake8 & pytest-linux/macOS

    Using PID to identify checkpoint files uniquely instead of Thread ID

    Removed test_diskckp.py. Tests moved to test_storage.py
  • Loading branch information
aluamorim committed Mar 9, 2021
0 parents commit debfce9
Show file tree
Hide file tree
Showing 50 changed files with 7,680 additions and 0 deletions.
1 change: 1 addition & 0 deletions .gitattributes
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
pyrevolve/_version.py export-subst
26 changes: 26 additions & 0 deletions .github/workflows/flake8.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,26 @@
name: Flake8

on:
push:
branches: [ master ]
pull_request:
branches: [ master ]

jobs:
flake8:

runs-on: ubuntu-latest

steps:
- uses: actions/checkout@v1
- name: Set up Python 3.7
uses: actions/setup-python@v1
with:
python-version: 3.7
- name: Install dependencies
run: |
python -m pip install --upgrade pip
pip install flake8
- name: Lint with flake8
run: |
flake8 --builtins=ArgumentError .
47 changes: 47 additions & 0 deletions .github/workflows/pytest-linux.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,47 @@
name: pytest-linux

on:
push:
branches: [ master ]
pull_request:
branches: [ master ]

jobs:
Build-linux:
runs-on: ${{ matrix.os }}
strategy:
matrix:
os: [ubuntu-18.04, ubuntu-20.04]
python-version: [3.8, 3.x]
compiler_v: [7,8,9]

steps:
- uses: actions/checkout@v2
- name: Set up Python ${{ matrix.python-version }}
uses: actions/setup-python@v2
with:
python-version: ${{ matrix.python-version }}
- name: Install dependencies
run: |
sudo add-apt-repository ppa:ubuntu-toolchain-r/test
sudo apt-get update
sudo apt-get install -y gcc-${{ matrix.compiler_v }} \
g++-${{ matrix.compiler_v }}
python -m pip install --upgrade pip
python -m pip install pytest numpy
if [ "${{ matrix.compiler_v }}" == '7' ];then
pip install -e .[compression]
else
pip install -e .
fi
env:
PYTHONPATH: $PWD
CC: gcc-${{ matrix.compiler_v }}
CXX: g++-${{ matrix.compiler_v }}
- name: Run examples
run: |
python examples/use_classic.py
python examples/use_modernised.py
- name: Test with pytest
run: py.test -vs tests

34 changes: 34 additions & 0 deletions .github/workflows/pytest-macOS.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,34 @@
name: pytest-macOS

on:
push:
branches: [ master ]
pull_request:
branches: [ master ]

jobs:
Build-macOS:
runs-on: macos-latest
strategy:
matrix:
python-version: [3.6-dev, 3.x]

steps:
- uses: actions/checkout@v2
- name: Set up Python ${{ matrix.python-version }}
uses: actions/setup-python@v2
with:
python-version: ${{ matrix.python-version }}
- name: Install dependencies
run: |
python -m pip install --upgrade pip
python -m pip install pytest numpy
pip install -e .
env:
PYTHONPATH: $PWD
- name: Run examples
run: |
python examples/use_classic.py
python examples/use_modernised.py
- name: Test with pytest
run: py.test -vs tests
26 changes: 26 additions & 0 deletions .github/workflows/pythonpublish.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,26 @@
name: Upload Python Package

on:
release:
types: [created]

jobs:
deploy:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v1
- name: Set up Python
uses: actions/setup-python@v1
with:
python-version: '3.x'
- name: Install dependencies
run: |
python -m pip install --upgrade pip
pip install setuptools wheel twine
- name: Build and publish
env:
TWINE_USERNAME: ${{ secrets.PYPI_USERNAME }}
TWINE_PASSWORD: ${{ secrets.PYPI_PASSWORD }}
run: |
python setup.py sdist
twine upload dist/*
91 changes: 91 additions & 0 deletions .travis.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,91 @@
sudo: false

language: python

matrix:
include:
- os: linux
python: "3.6"
addons:
apt:
sources:
- ubuntu-toolchain-r-test
packages:
- g++-4.9
- gcc-4.9
env:
- PYTHONPATH=$PWD
- CC=gcc-4.9
- CXX=g++-4.9

- os: linux
python: "3.6"
addons:
apt:
sources:
- ubuntu-toolchain-r-test
packages:
- g++-5
- gcc-5
env:
- PYTHONPATH=$PWD
- CC=gcc-5
- CXX=g++-5
install: pip install -e .[compression]

- os: linux
python: "3.6"
addons:
apt:
sources:
- ubuntu-toolchain-r-test
packages:
- g++-6
- gcc-6
env:
- PYTHONPATH=$PWD
- CC=gcc-6
- CXX=g++-6
- os: linux
python: "3.6"
addons:
apt:
sources:
- ubuntu-toolchain-r-test
packages:
- g++-7
- gcc-7
env:
- PYTHONPATH=$PWD
- CC=gcc-7
- CXX=g++-7
- os: linux
python: "3.6"
addons:
apt:
sources:
- ubuntu-toolchain-r-test
packages:
- g++-8
- gcc-8
env:
- PYTHONPATH=$PWD
- CC=gcc-8
- CXX=g++-8


- os: osx
osx_image: xcode8.3
python: "3.6-dev"
env: PYTHONPATH=$PWD

before_install: pip install -r requirements.txt

install:
- pip install -e .

script:
- flake8 .
- python examples/use_classic.py
- python examples/use_modernised.py
- py.test -vs tests/
Loading

0 comments on commit debfce9

Please sign in to comment.