Better driver needed #2
Comments
jhueckelheim
pushed a commit
that referenced
this issue
May 19, 2017
… now pass operators and a checkpoint object, pyrevolve does the rest.
|
The interface for pyrevolve has been completely overhauled as discussed in devitocodes/devito#216. Closing this issue. |
aluamorim
pushed a commit
to aluamorim/pyrevolve
that referenced
this issue
Mar 9, 2021
Wrapper around libadjoint Revolve implementations. Now contains online and multistage checkpointing. C++ code needs cleanup to remove warnings and missing functions. updated readme to reflect new interface removed function that is missing in C implementation from wrapper removed sections that only applied to the old wrapper improved wrapper: error handling, moved helper functions outside, as they are supposed to help the user find the right sizes /before/ creating an object added a convenience function that takes operators and handles the checkpointing functioning prototype for the devito-like Revolve wrapper cleanup in the wrapper, removed some print statements updated license, prettified code. nicer examples to illustrate Revolve slightly improved example code formatting flake8 is happy added check if symbols are passed as a dictionary, made Flake8 happy, made CRevolve a separete instance for each pyrevolve object. Rewrote API to improve separation of concerns. Fixes Issue devitocodes#2. We can now pass operators and a checkpoint object, pyrevolve does the rest. Update README.md updated readme to reflect new API Fixed compilation error. This may cause race conditions when more than one CRevolve object is used at the same time. add missing file removed file again Fix for python3 Add Operator ABC + change to new-style classes Add dtype to CheckpointStorage and add extra forward to align computations Update example to pass dtype Adding a requirements file and some basic travis testing. Still trying to get travis working Flake Hide the crevolve import in setup.py to give setuptools a chance to install Cython if not present Setup: `pyrevolve.crevolve` is not a package Test python 2 as well as 3 Import python2 style if python 3 style fails suppress uncalled for verbosity Fix issue 4: wrong argument order in multistage Add versioneer for automatic version numbering Add package metadata to setup.py Add pyx files to MANIFEST.in Make sure the provided action is something we understand Add basic testing Enable testing Flake Move utility functions out so they can be reused between tests Add blank line Ignore errors in versioneer Add initial infra for compression Separate the scheduler add blosc compression Add configurable compression params Some cleanup improving parameter handling Add tests for compression Allow parameters to be passed to the compression scheme Add ZFP compression scheme Flake Add a complete forward-backward test that can be expanded to test compression Add blosc to requirements Dirty hack to get travis to use zfp (because I'm too lazy to build a proper installation procedure for pyzfp) Fix flake issues Add contexttimer to reqs The complete forward-backward now works with zfp Changing APIs to hide the details of ByteStorage and compression All tests passing with new API All tests pass Add some logging + custom pickling functionality. This version works fine Some cleanup Add built-in profiler for pyrevolve Add counts to profiling Add more options to profiler Change to using copyto to copy Suppress console output from Revolve Change the API such that no excess data is copied over Add tools Profile storage Clean up by moving some code into separate files Adding missing files Some more changes Flake fixes Compression seems to work Enable testing on multiple GCC versions Still trying to get travis to accept osx+python combo Fix modern example Add requirements to Manifest Fix to cython version 2 Fix typo Remove pxd file Add back flake dependency to continue lint on CI Change the reference to tools Add static schedule generator Better directory structure and README Flakey changes Make travis test pip installation Update tests and travis build Update travis build Create pythonpublish.yml Update README Create CI workflow using GitHub Actions remove requirements and flake8 add flake8 & pytest-linux/macOS Added DiskStorage class to storage.py to store checkpoints to disk Adding DiskStorage class to storage.py file Created DiskRevolver and MemoryRevolver classes from Revolver. Renamed Revolver to BaseRevolver. Kept the name Revolver as MemoryRevolver for backward compatibility. The following changes were performed: 1 - Revolver class was split into: - BaseRevolver, which implements the the revolver FSM and receives a storage object as parameter; - DiskRevolver is an specialization of BaseRevolver that uses DiskStorage as storage method. - MemoryRevolver is an specialization of BaseRevolver that uses MemoryStorage as storage method. - The name Revolver was kept in the module for backward compatibility. If the user uses the name Revolver it reverts to MemoryRevolver as original. 2 - DiskStorage class implements the same interface as NumpyStorage, but saving data to a .dat file. The file path can be passed as parameter. 3 - Two test cases were added to validate memory and disk revolver classes. Removed bin files from tests/ Separated R/W file streams in storage.py. Re-wrote test_diskckp.py and test_memckp.py removing devito dependency. Fixed some flake8 related format issues Fixed some other flake8 related format issues Added thread Id to DiskStorage .dat file name; DiskStorage class now receives 'filedir' instead of 'filename'; BaseRevolver class was made abstract, forcing the user to instatiate either Memory or DiskRevolver objects. Added 'multi/singlefile' option to DiskRevovler and DiskStorage classes, enabling the user to store checkpoints in a single file or with one file per checkpoint; Added new tests for diskckp using multiple files. Using PID to identify checkpoint files uniquely instead of Thread ID Using PID to identify checkpoint files uniquely instead of Thread ID Removed 'print' calls; Added 'singlefile' parameter to test_diskckp.py to remove redundant tests Joined test_memckp.py and test_diskckp.py files using diskckp flag Move tests in test_diskckp.py to test_storage.py Removed bin files from tests/ Separated R/W file streams in storage.py. Re-wrote test_diskckp.py and test_memckp.py removing devito dependency. Added thread Id to DiskStorage .dat file name; DiskStorage class now receives 'filedir' instead of 'filename'; BaseRevolver class was made abstract, forcing the user to instatiate either Memory or DiskRevolver objects. Using PID to identify checkpoint files uniquely instead of Thread ID suppress uncalled for verbosity Fix issue 4: wrong argument order in multistage Add pyx files to MANIFEST.in Add blank line Separate the scheduler add blosc compression Add more options to profiler Change to using copyto to copy Add tools Still trying to get travis to accept osx+python combo Fix modern example Add requirements to Manifest Fix to cython version 2 Fix typo Remove pxd file Add back flake dependency to continue lint on CI Change the reference to tools Add static schedule generator Better directory structure and README Flakey changes Update travis build Create pythonpublish.yml Update README Removed bin files from tests/ Create CI workflow using GitHub Actions remove requirements and flake8 add flake8 & pytest-linux/macOS Using PID to identify checkpoint files uniquely instead of Thread ID Removed test_diskckp.py. Tests moved to test_storage.py
aluamorim
added a commit
to aluamorim/pyrevolve
that referenced
this issue
Mar 9, 2021
Wrapper around libadjoint Revolve implementations. Now contains online and multistage checkpointing. C++ code needs cleanup to remove warnings and missing functions.
updated readme to reflect new interface
removed function that is missing in C implementation from wrapper
removed sections that only applied to the old wrapper
improved wrapper: error handling, moved helper functions outside, as they are supposed to help the user find the right sizes /before/ creating an object
added a convenience function that takes operators and handles the checkpointing
functioning prototype for the devito-like Revolve wrapper
cleanup in the wrapper, removed some print statements
updated license, prettified code.
nicer examples to illustrate Revolve
slightly improved example code formatting
flake8 is happy
added check if symbols are passed as a dictionary, made Flake8 happy, made CRevolve a separete instance for each pyrevolve object.
Rewrote API to improve separation of concerns. Fixes Issue devitocodes#2. We can now pass operators and a checkpoint object, pyrevolve does the rest.
Update README.md
updated readme to reflect new API
Fixed compilation error. This may cause race conditions when more than one CRevolve object is used at the same time.
add missing file
removed file again
Fix for python3
Add Operator ABC + change to new-style classes
Add dtype to CheckpointStorage and add extra forward to align computations
Update example to pass dtype
Adding a requirements file and some basic travis testing.
Still trying to get travis working
Flake
Hide the crevolve import in setup.py to give setuptools a chance to install Cython if not present
Setup: is not a package
Test python 2 as well as 3
Import python2 style if python 3 style fails
suppress uncalled for verbosity
Fix issue 4: wrong argument order in multistage
Add versioneer for automatic version numbering
Add package metadata to setup.py
Add pyx files to MANIFEST.in
Make sure the provided action is something we understand
Add basic testing
Enable testing
Flake
Move utility functions out so they can be reused between tests
Add blank line
Ignore errors in versioneer
Add initial infra for compression
Separate the scheduler
add blosc compression
Add configurable compression params
Some cleanup
improving parameter handling
Add tests for compression
Allow parameters to be passed to the compression scheme
Add ZFP compression scheme
Flake
Add a complete forward-backward test that can be expanded to test compression
Add blosc to requirements
Dirty hack to get travis to use zfp (because I'm too lazy to build a proper installation procedure for pyzfp)
Fix flake issues
Add contexttimer to reqs
The complete forward-backward now works with zfp
Changing APIs to hide the details of ByteStorage and compression
All tests passing with new API
All tests pass
Add some logging + custom pickling functionality. This version works fine
Some cleanup
Add built-in profiler for pyrevolve
Add counts to profiling
Add more options to profiler
Change to using copyto to copy
Suppress console output from Revolve
Change the API such that no excess data is copied over
Add tools
Profile storage
Clean up by moving some code into separate files
Adding missing files
Some more changes
Flake fixes
Compression seems to work
Enable testing on multiple GCC versions
Still trying to get travis to accept osx+python combo
Fix modern example
Add requirements to Manifest
Fix to cython version 2
Fix typo
Remove pxd file
Add back flake dependency to continue lint on CI
Change the reference to tools
Add static schedule generator
Better directory structure and README
Flakey changes
Make travis test pip installation
Update tests and travis build
Update travis build
Create pythonpublish.yml
Update README
Create CI workflow using GitHub Actions
remove requirements and flake8
add flake8 & pytest-linux/macOS
Added DiskStorage class to storage.py to store checkpoints to disk
Adding DiskStorage class to storage.py file
Created DiskRevolver and MemoryRevolver classes from Revolver. Renamed Revolver to BaseRevolver. Kept the name Revolver as MemoryRevolver for backward compatibility.
The following changes were performed:
1 - Revolver class was split into:
- BaseRevolver, which implements the the revolver
FSM and receives a storage object as parameter;
- DiskRevolver is an specialization of BaseRevolver
that uses DiskStorage as storage method.
- MemoryRevolver is an specialization of BaseRevolver
that uses MemoryStorage as storage method.
- The name Revolver was kept in the module
for backward compatibility. If the user uses
the name Revolver it reverts to MemoryRevolver as original.
2 - DiskStorage class implements the same
interface as NumpyStorage, but saving data to
a .dat file. The file path can be passed as parameter.
3 - Two test cases were added to validate memory and disk revolver classes.
Removed bin files from tests/
Separated R/W file streams in storage.py. Re-wrote test_diskckp.py and test_memckp.py removing devito dependency.
Fixed some flake8 related format issues
Fixed some other flake8 related format issues
Added thread Id to DiskStorage .dat file name; DiskStorage class now receives 'filedir' instead of 'filename'; BaseRevolver class was made abstract, forcing the user to instatiate either Memory or DiskRevolver objects.
Added 'multi/singlefile' option to DiskRevovler and DiskStorage classes, enabling the user to store checkpoints in a single file or with one file per checkpoint; Added new tests for diskckp using multiple files.
Using PID to identify checkpoint files uniquely instead of Thread ID
Using PID to identify checkpoint files uniquely instead of Thread ID
Removed 'print' calls; Added 'singlefile' parameter to test_diskckp.py to remove redundant tests
Joined test_memckp.py and test_diskckp.py files using diskckp flag
Move tests in test_diskckp.py to test_storage.py
Removed bin files from tests/
Separated R/W file streams in storage.py. Re-wrote test_diskckp.py and test_memckp.py removing devito dependency.
Added thread Id to DiskStorage .dat file name; DiskStorage class now receives 'filedir' instead of 'filename'; BaseRevolver class was made abstract, forcing the user to instatiate either Memory or DiskRevolver objects.
Using PID to identify checkpoint files uniquely instead of Thread ID
suppress uncalled for verbosity
Fix issue 4: wrong argument order in multistage
Add pyx files to MANIFEST.in
Add blank line
Separate the scheduler
add blosc compression
Add more options to profiler
Change to using copyto to copy
Add tools
Still trying to get travis to accept osx+python combo
Fix modern example
Add requirements to Manifest
Fix to cython version 2
Fix typo
Remove pxd file
Add back flake dependency to continue lint on CI
Change the reference to tools
Add static schedule generator
Better directory structure and README
Flakey changes
Update travis build
Create pythonpublish.yml
Update README
Removed bin files from tests/
Create CI workflow using GitHub Actions
remove requirements and flake8
add flake8 & pytest-linux/macOS
Using PID to identify checkpoint files uniquely instead of Thread ID
Removed test_diskckp.py. Tests moved to test_storage.py
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
The current driver function is taken from the example code in the Revolve reference implementation. It does not do anything useful, and simply prints the actions that Revolve is requesting. It would be nice to have a driver that accepts <objects/function pointers/callbacks/something nice and pythonic> to actually do a useful computation, so that the user only needs to provide the worker functions and pyrevolve does all the rest. This is very simple in principle, but not yet sure how to do this nicely so that potential users can embed pyrevolve into their codes with minimum effort. To be discussed.
The text was updated successfully, but these errors were encountered: