Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Cannot find dlio_benchmark.py - errors #5

Open
laytonjbgmail opened this issue Aug 11, 2022 · 8 comments
Open

Cannot find dlio_benchmark.py - errors #5

laytonjbgmail opened this issue Aug 11, 2022 · 8 comments
Assignees
@hariharan-devarajan
Labels
bug Something isn't working

Comments

@laytonjbgmail
Copy link

Hello,

I install dlio using the following method

  1. Download source .zip file
  2. python setup.py install

The installation output is ,

$ python setup.py install
/home/laytonjb/anaconda3/lib/python3.9/site-packages/setuptools/dist.py:772: UserWarning: Usage of dash-separated 'description-file' will not be supported in future versions. Please use the underscore name 'description_file' instead
  warnings.warn(
running install
/home/laytonjb/anaconda3/lib/python3.9/site-packages/setuptools/command/install.py:34: SetuptoolsDeprecationWarning: setup.py install is deprecated. Use build and pip and other standards-based tools.
  warnings.warn(
/home/laytonjb/anaconda3/lib/python3.9/site-packages/setuptools/command/easy_install.py:144: EasyInstallDeprecationWarning: easy_install command is deprecated. Use build and pip and other standards-based tools.
  warnings.warn(
running bdist_egg
running egg_info
creating dlio.egg-info
writing dlio.egg-info/PKG-INFO
writing dependency_links to dlio.egg-info/dependency_links.txt
writing top-level names to dlio.egg-info/top_level.txt
writing manifest file 'dlio.egg-info/SOURCES.txt'
reading manifest file 'dlio.egg-info/SOURCES.txt'
adding license file 'LICENSE'
writing manifest file 'dlio.egg-info/SOURCES.txt'
installing library code to build/bdist.linux-x86_64/egg
running install_lib
warning: install_lib: 'build/lib' does not exist -- no Python modules to install

creating build
creating build/bdist.linux-x86_64
creating build/bdist.linux-x86_64/egg
creating build/bdist.linux-x86_64/egg/EGG-INFO
copying dlio.egg-info/PKG-INFO -> build/bdist.linux-x86_64/egg/EGG-INFO
copying dlio.egg-info/SOURCES.txt -> build/bdist.linux-x86_64/egg/EGG-INFO
copying dlio.egg-info/dependency_links.txt -> build/bdist.linux-x86_64/egg/EGG-INFO
copying dlio.egg-info/top_level.txt -> build/bdist.linux-x86_64/egg/EGG-INFO
zip_safe flag not set; analyzing archive contents...
creating dist
creating 'dist/dlio-0.0.1-py3.9.egg' and adding 'build/bdist.linux-x86_64/egg' to it
removing 'build/bdist.linux-x86_64/egg' (and everything under it)
Processing dlio-0.0.1-py3.9.egg
Removing /home/laytonjb/anaconda3/lib/python3.9/site-packages/dlio-0.0.1-py3.9.egg
Copying dlio-0.0.1-py3.9.egg to /home/laytonjb/anaconda3/lib/python3.9/site-packages
dlio 0.0.1 is already the active version in easy-install.pth

Installed /home/laytonjb/anaconda3/lib/python3.9/site-packages/dlio-0.0.1-py3.9.egg
Processing dependencies for dlio==0.0.1
Finished processing dependencies for dlio==0.0.1

I wanted to then run the command python ./dlio_benchmark.py but I cannot find the script. anywhere.

I tried going to the source to run dlio_benchmark.py, python3 ./dlio_benchmark.py and I get the following output,

$ python3 ./dlio_benchmark.py -h 
Traceback (most recent call last):
  File "/home/laytonjb/NVIDIA/ANL/dlio_benchmark-master/src/./dlio_benchmark.py", line 15, in <module>
    from src.common.enumerations import Profiler
ModuleNotFoundError: No module named 'src'

I apologize for missing something obvious - could someone help me?

BTW - Python3 3.9.12, horovod 0.25.0, Tensorflow 2.9.1, HDF5 3.7.0, numpy 1.21.6

Thank you.
@hariharan-devarajan
Copy link
Owner

I am sorry, you didnt miss anything, the setup.py has issues.

I would recommend you clone the repo and check the script https://github.com/hariharan-devarajan/dlio_benchmark/blob/master/scripts/cosmic.sh to see how to use it.

@laytonjbgmail
Copy link
Author

Thanks for the quick response. I cloned the repo to my workstation. Then I tried running setup.py again and got the following:

$ python ./setup.py install
/home/laytonjb/anaconda3/lib/python3.9/site-packages/setuptools/dist.py:772: UserWarning: Usage of dash-separated 'description-file' will not be supported in future versions. Please use the underscore name 'description_file' instead
  warnings.warn(
running install
/home/laytonjb/anaconda3/lib/python3.9/site-packages/setuptools/command/install.py:34: SetuptoolsDeprecationWarning: setup.py install is deprecated. Use build and pip and other standards-based tools.
  warnings.warn(
/home/laytonjb/anaconda3/lib/python3.9/site-packages/setuptools/command/easy_install.py:144: EasyInstallDeprecationWarning: easy_install command is deprecated. Use build and pip and other standards-based tools.
  warnings.warn(
running bdist_egg
running egg_info
creating dlio.egg-info
writing dlio.egg-info/PKG-INFO
writing dependency_links to dlio.egg-info/dependency_links.txt
writing top-level names to dlio.egg-info/top_level.txt
writing manifest file 'dlio.egg-info/SOURCES.txt'
reading manifest file 'dlio.egg-info/SOURCES.txt'
adding license file 'LICENSE'
writing manifest file 'dlio.egg-info/SOURCES.txt'
installing library code to build/bdist.linux-x86_64/egg
running install_lib
warning: install_lib: 'build/lib' does not exist -- no Python modules to install

creating build
creating build/bdist.linux-x86_64
creating build/bdist.linux-x86_64/egg
creating build/bdist.linux-x86_64/egg/EGG-INFO
copying dlio.egg-info/PKG-INFO -> build/bdist.linux-x86_64/egg/EGG-INFO
copying dlio.egg-info/SOURCES.txt -> build/bdist.linux-x86_64/egg/EGG-INFO
copying dlio.egg-info/dependency_links.txt -> build/bdist.linux-x86_64/egg/EGG-INFO
copying dlio.egg-info/top_level.txt -> build/bdist.linux-x86_64/egg/EGG-INFO
zip_safe flag not set; analyzing archive contents...
creating dist
creating 'dist/dlio-0.0.1-py3.9.egg' and adding 'build/bdist.linux-x86_64/egg' to it
removing 'build/bdist.linux-x86_64/egg' (and everything under it)
Processing dlio-0.0.1-py3.9.egg
Removing /home/laytonjb/anaconda3/lib/python3.9/site-packages/dlio-0.0.1-py3.9.egg
Copying dlio-0.0.1-py3.9.egg to /home/laytonjb/anaconda3/lib/python3.9/site-packages
dlio 0.0.1 is already the active version in easy-install.pth

Installed /home/laytonjb/anaconda3/lib/python3.9/site-packages/dlio-0.0.1-py3.9.egg
Processing dependencies for dlio==0.0.1
Finished processing dependencies for dlio==0.0.1

It had the same issues as my previous post.

I then tried going into the source on my workstation and try running dlio_benchmark.py. I got the following output that is the same as my previous post.

$ python ./dlio_benchmark.py -h 
Traceback (most recent call last):
  File "/home/laytonjb/NVIDIA/ANL/dlio_benchmark/src/./dlio_benchmark.py", line 15, in <module>
    from src.common.enumerations import Profiler
ModuleNotFoundError: No module named 'src'

Would it be better if I put all Python source files in one directory so that dlio_benchmark.py can find the module it needs? This seems drastic and very messy.

Thanks!

GG

@hariharan-devarajan
Copy link
Owner

you need to set the PYTHONPATH i believe.

I usually do this.

DLIO_ROOT=<Set your path to DLIO>
export PYTHONPATH=$DLIO_ROOT:$PYTHONPATH

The you can run dlio_benchmark.py

python ${DLIO_ROOT}/src/dlio_benchmark.py

@hariharan-devarajan
Copy link
Owner

Ok i tried a fresh environment and added the instructions into the README for your reference as well

git clone https://github.com/hariharan-devarajan/dlio_benchmark
cd dlio_benchmark/
python3 -m venv ./venv
source venv/bin/activate
pip install -r requirements.txt 
export PYTHONPATH=$PWD/src:$PYTHONPATH
python ./src/dlio_benchmark.py -h

@hariharan-devarajan
Copy link
Owner

I also fixed the setup.py now

you can install it by

python setup.py install
# this install dlio_benchmark as an executable.
dlio_benchmark -h

@hariharan-devarajan hariharan-devarajan self-assigned this Aug 12, 2022
@hariharan-devarajan hariharan-devarajan added the bug Something isn't working label Aug 12, 2022
@hariharan-devarajan
Copy link
Owner

We're u able to test this for you @laytonjbgmail .

@MartinHilgeman
Copy link

The dlio_benchmark executable does not work for me:

[martinh@zlogin01 dlio_benchmark]$ python3 setup.py install --prefix=$HOME/.local
running install
running bdist_egg
running egg_info
writing src/dlio.egg-info/PKG-INFO
writing dependency_links to src/dlio.egg-info/dependency_links.txt
writing entry points to src/dlio.egg-info/entry_points.txt
writing top-level names to src/dlio.egg-info/top_level.txt
reading manifest file 'src/dlio.egg-info/SOURCES.txt'
adding license file 'LICENSE'
writing manifest file 'src/dlio.egg-info/SOURCES.txt'
installing library code to build/bdist.linux-x86_64/egg
running install_lib
warning: install_lib: 'build/lib' does not exist -- no Python modules to install

creating build/bdist.linux-x86_64/egg
creating build/bdist.linux-x86_64/egg/EGG-INFO
copying src/dlio.egg-info/PKG-INFO -> build/bdist.linux-x86_64/egg/EGG-INFO
copying src/dlio.egg-info/SOURCES.txt -> build/bdist.linux-x86_64/egg/EGG-INFO
copying src/dlio.egg-info/dependency_links.txt -> build/bdist.linux-x86_64/egg/EGG-INFO
copying src/dlio.egg-info/entry_points.txt -> build/bdist.linux-x86_64/egg/EGG-INFO
copying src/dlio.egg-info/top_level.txt -> build/bdist.linux-x86_64/egg/EGG-INFO
zip_safe flag not set; analyzing archive contents...
creating 'dist/dlio-0.0.1-py3.7.egg' and adding 'build/bdist.linux-x86_64/egg' to it
removing 'build/bdist.linux-x86_64/egg' (and everything under it)
Processing dlio-0.0.1-py3.7.egg
Removing /home/martinh/.local/lib/python3.7/site-packages/dlio-0.0.1-py3.7.egg
Copying dlio-0.0.1-py3.7.egg to /home/martinh/.local/lib/python3.7/site-packages
dlio 0.0.1 is already the active version in easy-install.pth
Installing dlio_benchmark script to /home/martinh/.local/bin

Installed /home/martinh/.local/lib/python3.7/site-packages/dlio-0.0.1-py3.7.egg
Processing dependencies for dlio==0.0.1
Finished processing dependencies for dlio==0.0.1
Requirement already satisfied: horovod[tensorflow]>=0.19.5 in /home/martinh/.local/lib/python3.7/site-packages (from -r ./requirements.txt (line 1)) (0.26.1)
Requirement already satisfied: tensorflow>=2.2.0 in /home/martinh/.local/lib/python3.7/site-packages (from -r ./requirements.txt (line 2)) (2.2.0)
Requirement already satisfied: numpy>=1.19.1 in /home/martinh/.local/lib/python3.7/site-packages (from -r ./requirements.txt (line 3)) (1.19.1)
Requirement already satisfied: h5py~=2.10.0 in /home/martinh/.local/lib/python3.7/site-packages (from -r ./requirements.txt (line 4)) (2.10.0)
Requirement already satisfied: pandas>=1.1.3 in /home/martinh/.local/lib/python3.7/site-packages (from -r ./requirements.txt (line 5)) (1.3.5)
Requirement already satisfied: mpi4py>=3.1.3 in /home/martinh/.local/lib/python3.7/site-packages (from -r ./requirements.txt (line 6)) (3.1.3)
Requirement already satisfied: pyyaml in /home/martinh/.local/lib/python3.7/site-packages (from horovod[tensorflow]>=0.19.5->-r ./requirements.txt (line 1)) (6.0)
Requirement already satisfied: cffi>=1.4.0 in /home/martinh/.local/lib/python3.7/site-packages (from horovod[tensorflow]>=0.19.5->-r ./requirements.txt (line 1)) (1.15.1)
Requirement already satisfied: cloudpickle in /home/martinh/.local/lib/python3.7/site-packages (from horovod[tensorflow]>=0.19.5->-r ./requirements.txt (line 1)) (2.2.0)
Requirement already satisfied: packaging in /home/martinh/.local/lib/python3.7/site-packages (from horovod[tensorflow]>=0.19.5->-r ./requirements.txt (line 1)) (21.3)
Requirement already satisfied: psutil in /home/martinh/.local/lib/python3.7/site-packages (from horovod[tensorflow]>=0.19.5->-r ./requirements.txt (line 1)) (5.9.2)
Requirement already satisfied: termcolor>=1.1.0 in /home/martinh/.local/lib/python3.7/site-packages (from tensorflow>=2.2.0->-r ./requirements.txt (line 2)) (2.0.1)
Requirement already satisfied: google-pasta>=0.1.8 in /home/martinh/.local/lib/python3.7/site-packages (from tensorflow>=2.2.0->-r ./requirements.txt (line 2)) (0.2.0)
Requirement already satisfied: opt-einsum>=2.3.2 in /home/martinh/.local/lib/python3.7/site-packages (from tensorflow>=2.2.0->-r ./requirements.txt (line 2)) (3.3.0)
Requirement already satisfied: six>=1.12.0 in /cm/local/apps/python37/lib/python3.7/site-packages (from tensorflow>=2.2.0->-r ./requirements.txt (line 2)) (1.16.0)
Requirement already satisfied: absl-py>=0.7.0 in /home/martinh/.local/lib/python3.7/site-packages (from tensorflow>=2.2.0->-r ./requirements.txt (line 2)) (1.3.0)
Requirement already satisfied: grpcio>=1.8.6 in /home/martinh/.local/lib/python3.7/site-packages (from tensorflow>=2.2.0->-r ./requirements.txt (line 2)) (1.49.1)
Requirement already satisfied: tensorflow-estimator<2.3.0,>=2.2.0 in /home/martinh/.local/lib/python3.7/site-packages (from tensorflow>=2.2.0->-r ./requirements.txt (line 2)) (2.2.0)
Requirement already satisfied: astunparse==1.6.3 in /home/martinh/.local/lib/python3.7/site-packages (from tensorflow>=2.2.0->-r ./requirements.txt (line 2)) (1.6.3)
Requirement already satisfied: scipy==1.4.1 in /home/martinh/.local/lib/python3.7/site-packages (from tensorflow>=2.2.0->-r ./requirements.txt (line 2)) (1.4.1)
Requirement already satisfied: wrapt>=1.11.1 in /cm/local/apps/python37/lib/python3.7/site-packages (from tensorflow>=2.2.0->-r ./requirements.txt (line 2)) (1.12.1)
Requirement already satisfied: tensorboard<2.3.0,>=2.2.0 in /home/martinh/.local/lib/python3.7/site-packages (from tensorflow>=2.2.0->-r ./requirements.txt (line 2)) (2.2.2)
Requirement already satisfied: wheel>=0.26 in /home/martinh/.local/lib/python3.7/site-packages (from tensorflow>=2.2.0->-r ./requirements.txt (line 2)) (0.37.1)
Requirement already satisfied: gast==0.3.3 in /home/martinh/.local/lib/python3.7/site-packages (from tensorflow>=2.2.0->-r ./requirements.txt (line 2)) (0.3.3)
Requirement already satisfied: keras-preprocessing>=1.1.0 in /home/martinh/.local/lib/python3.7/site-packages (from tensorflow>=2.2.0->-r ./requirements.txt (line 2)) (1.1.2)
Requirement already satisfied: protobuf>=3.8.0 in /home/martinh/.local/lib/python3.7/site-packages (from tensorflow>=2.2.0->-r ./requirements.txt (line 2)) (3.20.1)
Requirement already satisfied: python-dateutil>=2.7.3 in /cm/local/apps/python37/lib/python3.7/site-packages (from pandas>=1.1.3->-r ./requirements.txt (line 5)) (2.8.2)
Requirement already satisfied: pytz>=2017.3 in /cm/local/apps/python37/lib/python3.7/site-packages (from pandas>=1.1.3->-r ./requirements.txt (line 5)) (2021.1)
Requirement already satisfied: pycparser in /home/martinh/.local/lib/python3.7/site-packages (from cffi>=1.4.0->horovod[tensorflow]>=0.19.5->-r ./requirements.txt (line 1)) (2.21)
Requirement already satisfied: werkzeug>=0.11.15 in /home/martinh/.local/lib/python3.7/site-packages (from tensorboard<2.3.0,>=2.2.0->tensorflow>=2.2.0->-r ./requirements.txt (line 2)) (2.2.2)
Requirement already satisfied: setuptools>=41.0.0 in /cm/local/apps/python37/lib/python3.7/site-packages (from tensorboard<2.3.0,>=2.2.0->tensorflow>=2.2.0->-r ./requirements.txt (line 2)) (57.4.0)
Requirement already satisfied: requests<3,>=2.21.0 in /cm/local/apps/python37/lib/python3.7/site-packages (from tensorboard<2.3.0,>=2.2.0->tensorflow>=2.2.0->-r ./requirements.txt (line 2)) (2.26.0)
Requirement already satisfied: markdown>=2.6.8 in /home/martinh/.local/lib/python3.7/site-packages (from tensorboard<2.3.0,>=2.2.0->tensorflow>=2.2.0->-r ./requirements.txt (line 2)) (3.4.1)
Requirement already satisfied: google-auth<2,>=1.6.3 in /home/martinh/.local/lib/python3.7/site-packages (from tensorboard<2.3.0,>=2.2.0->tensorflow>=2.2.0->-r ./requirements.txt (line 2)) (1.35.0)
Requirement already satisfied: google-auth-oauthlib<0.5,>=0.4.1 in /home/martinh/.local/lib/python3.7/site-packages (from tensorboard<2.3.0,>=2.2.0->tensorflow>=2.2.0->-r ./requirements.txt (line 2)) (0.4.6)
Requirement already satisfied: tensorboard-plugin-wit>=1.6.0 in /home/martinh/.local/lib/python3.7/site-packages (from tensorboard<2.3.0,>=2.2.0->tensorflow>=2.2.0->-r ./requirements.txt (line 2)) (1.8.1)
Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /home/martinh/.local/lib/python3.7/site-packages (from packaging->horovod[tensorflow]>=0.19.5->-r ./requirements.txt (line 1)) (3.0.9)
Requirement already satisfied: cachetools<5.0,>=2.0.0 in /home/martinh/.local/lib/python3.7/site-packages (from google-auth<2,>=1.6.3->tensorboard<2.3.0,>=2.2.0->tensorflow>=2.2.0->-r ./requirements.txt (line 2)) (4.2.4)
Requirement already satisfied: rsa<5,>=3.1.4 in /home/martinh/.local/lib/python3.7/site-packages (from google-auth<2,>=1.6.3->tensorboard<2.3.0,>=2.2.0->tensorflow>=2.2.0->-r ./requirements.txt (line 2)) (4.9)
Requirement already satisfied: pyasn1-modules>=0.2.1 in /home/martinh/.local/lib/python3.7/site-packages (from google-auth<2,>=1.6.3->tensorboard<2.3.0,>=2.2.0->tensorflow>=2.2.0->-r ./requirements.txt (line 2)) (0.2.8)
Requirement already satisfied: requests-oauthlib>=0.7.0 in /cm/local/apps/python37/lib/python3.7/site-packages (from google-auth-oauthlib<0.5,>=0.4.1->tensorboard<2.3.0,>=2.2.0->tensorflow>=2.2.0->-r ./requirements.txt (line 2)) (1.1.0)
Requirement already satisfied: importlib-metadata>=4.4 in /cm/local/apps/python37/lib/python3.7/site-packages (from markdown>=2.6.8->tensorboard<2.3.0,>=2.2.0->tensorflow>=2.2.0->-r ./requirements.txt (line 2)) (4.8.1)
Requirement already satisfied: certifi>=2017.4.17 in /cm/local/apps/python37/lib/python3.7/site-packages (from requests<3,>=2.21.0->tensorboard<2.3.0,>=2.2.0->tensorflow>=2.2.0->-r ./requirements.txt (line 2)) (2021.5.30)
Requirement already satisfied: idna<4,>=2.5 in /cm/local/apps/python37/lib/python3.7/site-packages (from requests<3,>=2.21.0->tensorboard<2.3.0,>=2.2.0->tensorflow>=2.2.0->-r ./requirements.txt (line 2)) (3.2)
Requirement already satisfied: charset-normalizer~=2.0.0 in /cm/local/apps/python37/lib/python3.7/site-packages (from requests<3,>=2.21.0->tensorboard<2.3.0,>=2.2.0->tensorflow>=2.2.0->-r ./requirements.txt (line 2)) (2.0.4)
Requirement already satisfied: urllib3<1.27,>=1.21.1 in /cm/local/apps/python37/lib/python3.7/site-packages (from requests<3,>=2.21.0->tensorboard<2.3.0,>=2.2.0->tensorflow>=2.2.0->-r ./requirements.txt (line 2)) (1.25.11)
Requirement already satisfied: MarkupSafe>=2.1.1 in /home/martinh/.local/lib/python3.7/site-packages (from werkzeug>=0.11.15->tensorboard<2.3.0,>=2.2.0->tensorflow>=2.2.0->-r ./requirements.txt (line 2)) (2.1.1)
Requirement already satisfied: typing-extensions>=3.6.4 in /cm/local/apps/python37/lib/python3.7/site-packages (from importlib-metadata>=4.4->markdown>=2.6.8->tensorboard<2.3.0,>=2.2.0->tensorflow>=2.2.0->-r ./requirements.txt (line 2)) (3.10.0.2)
Requirement already satisfied: zipp>=0.5 in /cm/local/apps/python37/lib/python3.7/site-packages (from importlib-metadata>=4.4->markdown>=2.6.8->tensorboard<2.3.0,>=2.2.0->tensorflow>=2.2.0->-r ./requirements.txt (line 2)) (3.5.0)
Requirement already satisfied: pyasn1<0.5.0,>=0.4.6 in /home/martinh/.local/lib/python3.7/site-packages (from pyasn1-modules>=0.2.1->google-auth<2,>=1.6.3->tensorboard<2.3.0,>=2.2.0->tensorflow>=2.2.0->-r ./requirements.txt (line 2)) (0.4.8)
Requirement already satisfied: oauthlib<3.0.0,>=2.1.0 in /cm/local/apps/python37/lib/python3.7/site-packages (from requests-oauthlib>=0.7.0->google-auth-oauthlib<0.5,>=0.4.1->tensorboard<2.3.0,>=2.2.0->tensorflow>=2.2.0->-r ./requirements.txt (line 2)) (2.1.0)
[martinh@zlogin01 dlio_benchmark]$ dlio_benchmark
Traceback (most recent call last):
  File "/home/martinh/.local/bin/dlio_benchmark", line 33, in <module>
    sys.exit(load_entry_point('dlio==0.0.1', 'console_scripts', 'dlio_benchmark')())
  File "/home/martinh/.local/bin/dlio_benchmark", line 25, in importlib_load_entry_point
    return next(matches).load()
  File "/cm/local/apps/python37/lib/python3.7/site-packages/importlib_metadata/__init__.py", line 194, in load
    module = import_module(match.group('module'))
  File "/cm/local/apps/python37/lib/python3.7/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1006, in _gcd_import
  File "<frozen importlib._bootstrap>", line 983, in _find_and_load
  File "<frozen importlib._bootstrap>", line 965, in _find_and_load_unlocked
ModuleNotFoundError: No module named 'dlio_benchmark'

The only way to get it to work is:

[martinh@zlogin01 dlio_benchmark]$ pwd
/home/martinh/work/dlio_benchmark
[martinh@zlogin01 dlio_benchmark]$ export PYTHONPATH=/home/martinh/work/dlio_benchmark
[martinh@zlogin01 dlio_benchmark]$ python ./src/dlio_benchmark.py -h
usage: dlio_benchmark.py [-h] [-f {tfrecord,hdf5,csv,npz,hdf5_opt}]
                         [-r {off,seed,random}] [-ms SHUFFLE_SIZE]
                         [-m {off,seed,random}] [-rt {memory,on_demand}]
                         [-fa {multi,shared,collective}] [-rl RECORD_LENGTH]
                         [-nf NUM_FILES] [-sf NUM_SAMPLES] [-bs BATCH_SIZE]
                         [-e EPOCHS] [-se SEED_CHANGE_EPOCH]
                         [-gd GENERATE_DATA] [-df DATA_FOLDER]
                         [-of OUTPUT_FOLDER] [-fp FILE_PREFIX]
                         [-go GENERATE_ONLY] [-k KEEP_FILES] [-p PROFILING]
                         [-l LOGDIR] [-s SEED] [-c CHECKPOINT]
                         [-sc STEPS_CHECKPOINT] [-ts TRANSFER_SIZE]
                         [-tr READ_THREADS] [-tc COMPUTATION_THREADS]
                         [-ct COMPUTATION_TIME] [-rp PREFETCH]
                         [-ps PREFETCH_SIZE] [-ec ENABLE_CHUNKING]
                         [-cs CHUNK_SIZE] [-co {none,gzip,lzf,bz2,zip,xz}]
                         [-cl COMPRESSION_LEVEL] [-d DEBUG]

DLIO Benchmark
...

@hariharan-devarajan
Copy link
Owner

We have migrated the repo to alcf located at https://github.com/argonne-lcf/dlio_benchmark. Can you please check that repo and if issue persists please raise the issue there. Sorry for the inconvenience.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@hariharan-devarajan hariharan-devarajan

Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@MartinHilgeman @hariharan-devarajan @laytonjbgmail