Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[pull] master from pangeo-data:master #2

Open
wants to merge 205 commits into
base: master
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
205 commits
Select commit Hold shift + click to select a range
28576a3
Add pre-commit hooks
andersy005 Apr 23, 2019
442a3a6
Add function to generate random timeseries
andersy005 Apr 23, 2019
3c713d8
Add conda env file
andersy005 Apr 23, 2019
323c657
Add computation operations
andersy005 Apr 23, 2019
0ccb759
Add utilities
andersy005 Apr 23, 2019
045998b
Add cluster wait function
andersy005 Apr 23, 2019
bb6a6a0
Add setup class for running benchmarks
andersy005 Apr 23, 2019
c3b41cb
add initial run
andersy005 Apr 23, 2019
1e99d8c
Merge pull request #4 from andersy005/master
jmunroe Apr 23, 2019
1503ea0
add new results
andersy005 May 5, 2019
87a2686
add new results
andersy005 May 5, 2019
cce5602
Add plots for results
andersy005 May 5, 2019
3d1e2c9
Update plots
andersy005 May 5, 2019
797e8d7
Merge pull request #5 from andersy005/master
andersy005 May 5, 2019
7419c75
add new results
andersy005 May 13, 2019
dfd3c29
Merge pull request #8 from andersy005/master
andersy005 May 13, 2019
52d5251
Add chunking_scheme parameter
andersy005 May 22, 2019
7ae3bea
remove notebook
andersy005 May 22, 2019
ba72999
add more runs
andersy005 May 22, 2019
6527793
add results for more runs
andersy005 May 23, 2019
958c4a3
add more results
andersy005 May 23, 2019
28e2883
more runs
andersy005 May 23, 2019
2efb057
add runs for 512 MB chunk size
andersy005 May 23, 2019
dd59420
add weak scaling plots
andersy005 May 23, 2019
e762ed7
add post build script
andersy005 May 23, 2019
8b7f002
Merge pull request #9 from andersy005/master
andersy005 May 23, 2019
895f17b
update plots
andersy005 May 23, 2019
d4f0aac
Merge pull request #10 from andersy005/master
andersy005 May 23, 2019
91d744b
modification to run benchmark on 'hal'
Jun 7, 2019
daf6edb
modification to run benchmark on 'hal'
Jun 7, 2019
fef104d
initial benchmark result on 'hal'
Jun 7, 2019
7ff9813
conda enviroment used to plot on my mac'
Jun 7, 2019
262d695
update yml file for conda enviroment
Jun 13, 2019
f57cb68
Delete pangeo_yml
tinaok Jun 13, 2019
c425a9f
modified pangeobench.yml
Jun 13, 2019
e900986
Merge branch 'master' of https://github.com/tinaok/benchmarking
Jun 13, 2019
b610e49
update pangeobench.yml
Jun 13, 2019
38dbab2
modified: pangeobench.yml
Jun 13, 2019
fdcd0fd
modified: pangeobench.yml
Jun 13, 2019
3223a48
modified: pangeobench.yml
Jun 14, 2019
bb63777
modification on util.py
Jun 14, 2019
5841dd6
add machine variable for plot_result, and created plot_results_machi…
Jun 20, 2019
cae3efa
Merge pull request #11 from tinaok/master
Aug 1, 2019
d5762c0
Adding 'maxmemory per node, maxcore per node' for specifying HPC clus…
Aug 1, 2019
b412495
Modification to include machine specific configuration.
Aug 9, 2019
c954295
hal benchmarking result: by blocking hal one full node=24c 128g memory
Aug 13, 2019
ed60d9f
change name hal bench
Aug 13, 2019
60e9ed2
Adding bug fix : ensuring that each worker process only have one threads
Aug 14, 2019
f6e792f
add more comments in utils.py
tinaok Aug 14, 2019
77920da
Merge pull request #12 from tinaok/master
Aug 14, 2019
91035e7
Idial test case to mesure the slowdown only due to the communication …
tinaok Aug 21, 2019
625fb21
update plot and comments
tinaok Aug 21, 2019
fbca85d
adding chunk_per_worker as variable for a test (before it was only po…
tinaok Aug 22, 2019
56799f9
new test series on hal
tinaok Aug 22, 2019
90ed9dc
still test version of plotting, but have subplot function ready
Aug 22, 2019
a471c16
Merge branch 'master' of https://github.com/tinaok/benchmarking
Aug 22, 2019
f470a9d
created cheyenne.yaml for benchmarking in cheyenne
tinaok Aug 23, 2019
de960e7
Merge branch 'master' of https://github.com/tinaok/benchmarking
tinaok Aug 23, 2019
662781d
draft commit, before weekend.
Aug 23, 2019
838d60e
Add idias for adding IO bench
tinaok Aug 25, 2019
afcc7f7
Updating idias for ops.py for adding IO bench
tinaok Aug 25, 2019
e4c598d
deleting unused test results and analysis
Aug 28, 2019
63b8d34
Cleaning up hal test configurations
tinaok Aug 28, 2019
47cb3e8
cleaning notebook for submitting test jobs on hal
Aug 28, 2019
ff3755e
cleaning notebook for submitting test jobs on hal
tinaok Aug 28, 2019
16dc735
Merge branch 'master' of https://github.com/tinaok/benchmarking
tinaok Aug 28, 2019
1fc9fa8
cleaning the plots
tinaok Aug 28, 2019
0fb1f4c
adding new plotting notebook. beta version.
tinaok Aug 28, 2019
3066067
lazy bug fix for issue 14 and 15
tinaok Aug 28, 2019
cb1609e
adding big chunk with small nodes tests to make figures
tinaok Aug 30, 2019
88c15b4
some more computing result beeing added
tinaok Aug 30, 2019
5d8215d
addding some more hal computation results
tinaok Sep 5, 2019
1f2e229
take out crashed result, results-halrerun/compute_study_2019-09-03_1…
tinaok Sep 5, 2019
923651a
Merge pull request #13 from tinaok/master
andersy005 Sep 7, 2019
5a9e182
Rearrange outputs
andersy005 Sep 7, 2019
6dc4824
Use logging module for logging instead of print
andersy005 Sep 7, 2019
8b1026c
Fixes #14, Fixes #15
andersy005 Sep 7, 2019
e06e946
Add command line utility
andersy005 Sep 7, 2019
cae802f
temporal_mean -> spatial_mean, global_mean -> temporal_mean
andersy005 Sep 7, 2019
71a4928
Update environment and README
andersy005 Sep 7, 2019
bfbe553
Fix typo
andersy005 Sep 7, 2019
0af827d
Merge pull request #19 from andersy005/rearange-output
andersy005 Sep 7, 2019
18d0687
Merge branch 'master' of github.com:pangeo-data/benchmarking into cli
andersy005 Sep 7, 2019
4291ed8
Fix typo
andersy005 Sep 7, 2019
aa683fb
Merge pull request #20 from andersy005/cli
andersy005 Sep 7, 2019
5ff3e08
Fix pandas date out of range
andersy005 Sep 7, 2019
09e4517
Add Cheyenne results
andersy005 Sep 7, 2019
da37504
add wrangler results
andersy005 Sep 7, 2019
215450b
Merge pull request #22 from andersy005/master
andersy005 Sep 7, 2019
b3f7280
Merge branch 'master' of github.com:andersy005/benchmarking into wran…
andersy005 Sep 7, 2019
eb2d1f7
update wrangler config
andersy005 Sep 7, 2019
3ef7fb2
Update README.md
andersy005 Sep 7, 2019
b927680
Merge pull request #23 from andersy005/wrangler
andersy005 Sep 7, 2019
a038485
Merge pull request #1 from pangeo-data/master
tinaok Sep 9, 2019
c0d8341
adding some more benchmark config for cheyenne and reorganising direc…
tinaok Sep 10, 2019
79db7d7
add new cheyenne results
andersy005 Sep 11, 2019
2eb6717
Merge pull request #24 from tinaok/master
tinaok Sep 11, 2019
bfbba36
Merge pull request #2 from andersy005/more-cheyenne-computation
tinaok Sep 11, 2019
970070a
Merge branch 'master' of github.com:pangeo-data/benchmarking into mor…
andersy005 Sep 11, 2019
f4214d4
Use a single config file
andersy005 Sep 11, 2019
f3f4ef3
Merge pull request #25 from andersy005/more-cheyenne-computation
andersy005 Sep 11, 2019
feb5d03
Fixes #27
andersy005 Sep 11, 2019
696b3d5
Merge pull request #28 from andersy005/patch/operation-renaming
andersy005 Sep 12, 2019
5951eb6
Merge pull request #3 from pangeo-data/master
tinaok Sep 12, 2019
b598095
commiting notebooks used to create figures for the paper
tinaok Oct 10, 2019
09d1943
Merge pull request #31 from tinaok/master
tinaok Oct 10, 2019
78a1486
updating config files for hal
tinaok Oct 10, 2019
93f3b28
update jupyter notebook running file for hal
tinaok Oct 10, 2019
e0e2b61
updating hal.ipynb for repeated benchmark on hal
tinaok Oct 10, 2019
6fd5d40
updating script for making figures for hal
tinaok Oct 10, 2019
9a4026a
first results from hal with newer benchmarking method
tinaok Oct 10, 2019
da8e955
Merge branch 'master' of https://github.com/tinaok/benchmarking
tinaok Oct 10, 2019
bcc89bd
intermediate results hal
tinaok Oct 10, 2019
ac0e983
adding some new result with hourly benchmarks on hal
tinaok Oct 10, 2019
bdbd307
updating analysis on hal with 1H & 1D datasets
tinaok Oct 10, 2019
9d90ebb
adding results
tinaok Oct 10, 2019
e4a5510
add more results with 1H , weighted computation from HAL
tinaok Oct 11, 2019
96c82fb
add more results with 1H , weighted computation from HAL
tinaok Oct 11, 2019
d801bb5
more results
tinaok Oct 11, 2019
d0d4f6d
Merge branch 'master' of https://github.com/tinaok/benchmarking
tinaok Oct 11, 2019
fd5a218
Move daily data sets computing result done on hal in a directory: res…
tinaok Oct 11, 2019
f7b02bf
keep the old name of directory for original hal's computational resul…
tinaok Oct 11, 2019
dc5e4cc
reverifying figures for camera ready proof.
tinaok Oct 11, 2019
720d36c
Update readme
tinaok Oct 11, 2019
e9a4923
adding empty jupyter notebook for running benchmark from jupyter note…
tinaok Oct 11, 2019
b67568f
Merge pull request #32 from tinaok/master
tinaok Oct 11, 2019
aec75ef
issue #33
tinaok Oct 11, 2019
2c1605b
Merge pull request #34 from tinaok/master
tinaok Oct 11, 2019
f5518be
adding benchmarking results using daily data set (with coefficients i…
tinaok Oct 11, 2019
4a5466c
cleaning script for issue #37
tinaok Oct 11, 2019
fd5c7ba
fixing issue #37 and re-plot the result.
tinaok Oct 11, 2019
db10b12
Merge pull request #38 from tinaok/master
tinaok Oct 11, 2019
97a0d25
Update README.md
tinaok Jan 16, 2020
334e599
test
halehawk Feb 20, 2020
6760a95
test again
halehawk Feb 20, 2020
500cabd
see if upstream set
halehawk Feb 20, 2020
5e95e83
test hook with tab
halehawk Feb 20, 2020
b7f6597
reomve test_hook.py
halehawk Feb 20, 2020
3ddcce3
remove test.md
halehawk Feb 20, 2020
cf3da94
add pip package setup.py
halehawk Mar 23, 2020
921fec1
Update setup.py
halehawk Mar 23, 2020
fafc0f1
add install_req
halehawk Mar 30, 2020
3768783
Update setup.py
halehawk Mar 31, 2020
21cb587
Update setup.py
halehawk Mar 31, 2020
b15bf6b
Make the repository pip installable (#40)
halehawk Mar 31, 2020
7e8ed9c
Create main.yaml
halehawk Apr 1, 2020
a82038e
Merge branch 'pip-pack'
halehawk Apr 1, 2020
5ed6207
added actions/main.yaml
halehawk Apr 1, 2020
2c4d8a1
modified pre-commit
halehawk Apr 1, 2020
397d8b4
removed blank line in pre-commit-hook
halehawk Apr 1, 2020
36dc5ea
import find_package
halehawk Apr 1, 2020
ebbbe63
resort imports
halehawk Apr 1, 2020
9d5e2aa
resort imports
halehawk Apr 1, 2020
9f145db
reorder imports
halehawk Apr 1, 2020
15068cd
reorder imports
halehawk Apr 1, 2020
3790e86
reorder sort
halehawk Apr 1, 2020
5643be5
add blank line
halehawk Apr 1, 2020
2ea5bec
add comments
halehawk Apr 1, 2020
224ef81
add click as third party
halehawk Apr 1, 2020
f21b5ac
changed first party module
halehawk Apr 1, 2020
14603e1
reorder import
halehawk Apr 1, 2020
c917e3b
reorder import
halehawk Apr 2, 2020
4c79c13
switch Python3 toPython
halehawk Apr 2, 2020
0a08e48
reorder
halehawk Apr 2, 2020
6069388
reorder
halehawk Apr 2, 2020
5687583
add one line
halehawk Apr 2, 2020
0511801
remove first party
halehawk Apr 2, 2020
0f7c0ad
add comments
halehawk Apr 2, 2020
50e67e1
install seed-isort
halehawk Apr 2, 2020
af848f2
add requirements to solve third party click
halehawk Apr 2, 2020
1e0b56d
install requirements.txt in main.yaml
halehawk Apr 2, 2020
ecd07bd
tested isort:skip
halehawk Apr 2, 2020
1f0e8cc
Create pythonpublish.yml
halehawk Apr 3, 2020
4636f56
add click
halehawk Apr 6, 2020
96673a2
Create pythonpublish2.yml
halehawk Apr 6, 2020
300aa8c
removed isort:skip
halehawk Apr 6, 2020
3fa4bdc
merged actions
halehawk Apr 6, 2020
223d600
testing publish
halehawk Apr 6, 2020
2071853
upload to test_pypi
halehawk Apr 6, 2020
6d0a277
change classifiers
halehawk Apr 6, 2020
f3c33b7
remove pythonpublish2.yml
halehawk Apr 6, 2020
03970a9
modify pythonpublish.yml
halehawk Apr 6, 2020
8ff10c0
Create code style action and python publish action (#41)
halehawk Apr 13, 2020
40469ac
add write and read file
halehawk May 15, 2020
9906066
update with master
halehawk May 15, 2020
2ebac75
update datasets.py
halehawk May 15, 2020
df4cb14
remove f-string placeholders
halehawk May 15, 2020
b82f0ba
added openfile, writefile, readfile, deletefile
halehawk Jun 3, 2020
8ecba4a
modify code styles
halehawk Jun 3, 2020
efd5c3a
changed filename
halehawk Jun 15, 2020
dc199cc
add plots
halehawk Jun 23, 2020
a14def0
paper version
halehawk Aug 24, 2020
28a8bad
Update README.md
halehawk Aug 24, 2020
0e97acf
Update README.md
halehawk Aug 24, 2020
8034e43
consolidate openfile
halehawk Aug 25, 2020
972e947
added plotting notebooks
halehawk Oct 2, 2020
dd42a1f
save to html instead of png
halehawk Oct 4, 2020
f1aff63
tidy the codes
halehawk Jan 13, 2021
4f5f033
added version and env_export functions, and upload files to s3
halehawk Jan 27, 2021
42f55fd
fixed black format error
halehawk Jan 27, 2021
d978f1c
test publish again
halehawk Jan 27, 2021
7e2e456
update README with subcommands
halehawk Jan 27, 2021
a199f32
remove conda_env_export.py
halehawk Jan 29, 2021
d955676
add guides on README.md
halehawk Jan 29, 2021
909a191
Merge pull request #44 from NCAR/master
Feb 2, 2021
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
The table of contents is too big for display.
Diff view
Diff view
  •  
  •  
  •  
65 changes: 65 additions & 0 deletions .github/workflows/main.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,65 @@
name: code-style

on:
push:
branches: "*"
pull_request:
branches: master



jobs:

lint-black:
name: black-formatter
runs-on: ubuntu-latest
steps:
- name: Checkout
uses: actions/checkout@v1
- name: Set up Python 3.6
uses: actions/setup-python@v1
with:
python-version: 3.6
- name: Install Black
run: |
python -m pip install black
- name: Black Code Formatter
run: |
black --check --line-length 100 --skip-string-normalization .

lint-flake8:
name: flake8
runs-on: ubuntu-latest
steps:
- name: Checkout
uses: actions/checkout@v1
- name: Set up Python 3.6
uses: actions/setup-python@v1
with:
python-version: 3.6
- name: Install Flake8
run: |
python -m pip install flake8
- name: Flake8 check
run: |
flake8 .

lint-isort:
name: isort
runs-on: ubuntu-latest
steps:
- name: Checkout
uses: actions/checkout@v1
- name: Set up Python 3.6
uses: actions/setup-python@v1
with:
python-version: 3.6
- name: Install isort
run: |
python -m pip install isort
- name: Install dependency
run: |
python -m pip install -r requirements.txt
- name: isort check
run: |
isort --recursive --check-only .
38 changes: 38 additions & 0 deletions .github/workflows/pythonpublish.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,38 @@
# This is a basic workflow to help you get started with Actions

name: Upload Python Package

# Controls when the action will run. Triggers the workflow on push or pull request
# events but only for the master branch
on:
release:
types: [created]

# A workflow run is made up of one or more jobs that can run sequentially or in parallel
jobs:
# This workflow contains a single job called "deploy"
deploy:
# The type of runner that the job will run on
runs-on: ubuntu-latest

# Steps represent a sequence of tasks that will be executed as part of the job
steps:
# Checks-out your repository under $GITHUB_WORKSPACE, so your job can access it
- uses: actions/checkout@v1

# Runs a single command using the runners shell
- name: Set up Python
uses: actions/setup-python@v1
with:
python-version: '3.x'
- name: Install dependencies
run: |
python -m pip install --upgrade pip
pip install setuptools setuptools-scm wheel twine
- name: Build and publish
env:
TWINE_USERNAME: ${{ secrets.PYPI_USERNAME }}
TWINE_PASSWORD: ${{ secrets.PYPI_PASSWORD }}
run: |
python setup.py sdist bdist_wheel
twine upload dist/*
106 changes: 106 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
@@ -0,0 +1,106 @@
# Byte-compiled / optimized / DLL files
__pycache__/
*.py[cod]
*$py.class

# C extensions
*.so

# Distribution / packaging
.Python
build/
develop-eggs/
dist/
downloads/
eggs/
.eggs/
lib/
lib64/
parts/
sdist/
var/
wheels/
*.egg-info/
.installed.cfg
*.egg
MANIFEST

# PyInstaller
# Usually these files are written by a python script from a template
# before PyInstaller builds the exe, so as to inject date/other infos into it.
*.manifest
*.spec

# Installer logs
pip-log.txt
pip-delete-this-directory.txt

# Unit test / coverage reports
htmlcov/
.tox/
.coverage
.coverage.*
.cache
nosetests.xml
coverage.xml
*.cover
.hypothesis/
.pytest_cache/

# Translations
*.mo
*.pot

# Django stuff:
*.log
local_settings.py
db.sqlite3

# Flask stuff:
instance/
.webassets-cache

# Scrapy stuff:
.scrapy

# Sphinx documentation
docs/_build/

# PyBuilder
target/

# Jupyter Notebook
.ipynb_checkpoints

# pyenv
.python-version

# celery beat schedule file
celerybeat-schedule

# SageMath parsed files
*.sage.py

# Environments
.env
.venv
env/
venv/
ENV/
env.bak/
venv.bak/

# Spyder project settings
.spyderproject
.spyproject

# Rope project settings
.ropeproject

# mkdocs documentation
/site

# mypy
.mypy_cache/
_build/
.vscode/
36 changes: 36 additions & 0 deletions .pre-commit-config.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,36 @@
repos:

- repo: https://github.com/pre-commit/pre-commit-hooks
rev: v2.3.0
hooks:
- id: trailing-whitespace
- id: end-of-file-fixer
- id: check-docstring-first
- id: check-added-large-files
args: ['--maxkb=5000']
- id: check-json
- id: check-yaml
- id: double-quote-string-fixer

- repo: https://github.com/ambv/black
rev: 19.3b0
hooks:
- id: black
args: ["--line-length", "100", "--skip-string-normalization"]


- repo: https://github.com/pre-commit/pre-commit-hooks
rev: v2.3.0
hooks:
- id: flake8
args: ["--max-line-length", "100"]

- repo: https://github.com/asottile/seed-isort-config
rev: v2.1.0
hooks:
- id: seed-isort-config

- repo: https://github.com/pre-commit/mirrors-isort
rev: v4.3.21
hooks:
- id: isort
Loading