diff --git a/README.md b/README.md
index 0385cfa7d3..455b9d5a35 100755
--- a/README.md
+++ b/README.md
@@ -39,7 +39,7 @@ It consists of the following sub-projects:
* [Collective Knowledge platform (MLCommons CK playground)](platform)
aggregates [reproducible experiments](https://access.cknowledge.org/playground/?action=experiments)
in the CM format, connects academia and industry to
- [organize reproducibility and optimization challenges]( https://github.com/mlcommons/ck/tree/master/cm-mlops/challenge ),
+ [organize reproducibility, replicability and optimization challenges]( https://github.com/mlcommons/ck/tree/master/cm-mlops/challenge ),
and help developers and users select Pareto-optimal end-to-end applications and systems based on their requirements and constraints
(cost, performance, power consumption, accuracy, etc).
diff --git a/cm-mlops/automation/cache/README.md b/cm-mlops/automation/cache/README.md
index 597dd08875..fc5b065aa4 100644
--- a/cm-mlops/automation/cache/README.md
+++ b/cm-mlops/automation/cache/README.md
@@ -1,4 +1,4 @@
-*This README is automatically generated - don't edit! Use `README-extra.md` for extra notes!*
+*This README is automatically generated - don't edit! See [extra README](README-extra.md) for extra notes!*
### Automation actions
@@ -44,4 +44,4 @@
### Maintainers
-* [Open MLCommons taskforce on automation and reproducibility](https://github.com/mlcommons/ck/blob/master/docs/mlperf-education-workgroup.md)
\ No newline at end of file
+* [Open MLCommons taskforce on automation and reproducibility](https://cKnowledge.org/mlcommons-taskforce)
\ No newline at end of file
diff --git a/cm-mlops/automation/challenge/README.md b/cm-mlops/automation/challenge/README.md
new file mode 100644
index 0000000000..2db03e8b16
--- /dev/null
+++ b/cm-mlops/automation/challenge/README.md
@@ -0,0 +1,27 @@
+*This README is automatically generated - don't edit! Use `README-extra.md` for extra notes!*
+
+### Automation actions
+
+#### test
+
+ * CM CLI: ```cm test challenge``` ([add flags (dict keys) from this API](https://github.com/mlcommons/ck/tree/master/cm-mlops/automation/challenge/module.py#L15))
+ * CM CLI with UID: ```cm test challenge,3d84abd768f34e08``` ([add flags (dict keys) from this API](https://github.com/mlcommons/ck/tree/master/cm-mlops/automation/challenge/module.py#L15))
+ * CM Python API:
+ ```python
+ import cmind
+
+ r=cm.access({
+ 'action':'test'
+ 'automation':'challenge,3d84abd768f34e08'
+ 'out':'con'
+ ```
+ [add keys from this API](https://github.com/mlcommons/ck/tree/master/cm-mlops/automation/challenge/module.py#L15)
+ ```python
+ })
+ if r['return']>0:
+ print(r['error'])
+ ```
+
+### Maintainers
+
+* [Open MLCommons taskforce on automation and reproducibility](https://cKnowledge.org/mlcommons-taskforce)
\ No newline at end of file
diff --git a/cm-mlops/automation/contributor/README.md b/cm-mlops/automation/contributor/README.md
new file mode 100644
index 0000000000..c75d00f83c
--- /dev/null
+++ b/cm-mlops/automation/contributor/README.md
@@ -0,0 +1,27 @@
+*This README is automatically generated - don't edit! Use `README-extra.md` for extra notes!*
+
+### Automation actions
+
+#### test
+
+ * CM CLI: ```cm test contributor``` ([add flags (dict keys) from this API](https://github.com/mlcommons/ck/tree/master/cm-mlops/automation/contributor/module.py#L15))
+ * CM CLI with UID: ```cm test contributor,68eae17b590d4f8f``` ([add flags (dict keys) from this API](https://github.com/mlcommons/ck/tree/master/cm-mlops/automation/contributor/module.py#L15))
+ * CM Python API:
+ ```python
+ import cmind
+
+ r=cm.access({
+ 'action':'test'
+ 'automation':'contributor,68eae17b590d4f8f'
+ 'out':'con'
+ ```
+ [add keys from this API](https://github.com/mlcommons/ck/tree/master/cm-mlops/automation/contributor/module.py#L15)
+ ```python
+ })
+ if r['return']>0:
+ print(r['error'])
+ ```
+
+### Maintainers
+
+* [Open MLCommons taskforce on automation and reproducibility](https://cKnowledge.org/mlcommons-taskforce)
\ No newline at end of file
diff --git a/cm-mlops/automation/docker/README.md b/cm-mlops/automation/docker/README.md
index 140d8693db..c6ef9a3842 100644
--- a/cm-mlops/automation/docker/README.md
+++ b/cm-mlops/automation/docker/README.md
@@ -24,4 +24,4 @@
### Maintainers
-* [Open MLCommons taskforce on automation and reproducibility](https://github.com/mlcommons/ck/blob/master/docs/mlperf-education-workgroup.md)
\ No newline at end of file
+* [Open MLCommons taskforce on automation and reproducibility](https://cKnowledge.org/mlcommons-taskforce)
\ No newline at end of file
diff --git a/cm-mlops/automation/experiment/README.md b/cm-mlops/automation/experiment/README.md
index e5124f62f6..e1257d1fb8 100644
--- a/cm-mlops/automation/experiment/README.md
+++ b/cm-mlops/automation/experiment/README.md
@@ -64,4 +64,4 @@
### Maintainers
-* [Open MLCommons taskforce on automation and reproducibility](https://github.com/mlcommons/ck/blob/master/docs/mlperf-education-workgroup.md)
\ No newline at end of file
+* [Open MLCommons taskforce on automation and reproducibility](https://cKnowledge.org/mlcommons-taskforce)
\ No newline at end of file
diff --git a/cm-mlops/automation/project/README.md b/cm-mlops/automation/project/README.md
index 29e9bf961d..e684ac7ade 100644
--- a/cm-mlops/automation/project/README.md
+++ b/cm-mlops/automation/project/README.md
@@ -24,4 +24,4 @@
### Maintainers
-* [Open MLCommons taskforce on automation and reproducibility](https://github.com/mlcommons/ck/blob/master/docs/mlperf-education-workgroup.md)
\ No newline at end of file
+* [Open MLCommons taskforce on automation and reproducibility](https://cKnowledge.org/mlcommons-taskforce)
\ No newline at end of file
diff --git a/cm-mlops/automation/report/README.md b/cm-mlops/automation/report/README.md
new file mode 100644
index 0000000000..6f2f966963
--- /dev/null
+++ b/cm-mlops/automation/report/README.md
@@ -0,0 +1,27 @@
+*This README is automatically generated - don't edit! Use `README-extra.md` for extra notes!*
+
+### Automation actions
+
+#### test
+
+ * CM CLI: ```cm test report``` ([add flags (dict keys) from this API](https://github.com/mlcommons/ck/tree/master/cm-mlops/automation/report/module.py#L15))
+ * CM CLI with UID: ```cm test report,6462ecdba2054467``` ([add flags (dict keys) from this API](https://github.com/mlcommons/ck/tree/master/cm-mlops/automation/report/module.py#L15))
+ * CM Python API:
+ ```python
+ import cmind
+
+ r=cm.access({
+ 'action':'test'
+ 'automation':'report,6462ecdba2054467'
+ 'out':'con'
+ ```
+ [add keys from this API](https://github.com/mlcommons/ck/tree/master/cm-mlops/automation/report/module.py#L15)
+ ```python
+ })
+ if r['return']>0:
+ print(r['error'])
+ ```
+
+### Maintainers
+
+* [Open MLCommons taskforce on automation and reproducibility](https://cKnowledge.org/mlcommons-taskforce)
\ No newline at end of file
diff --git a/cm-mlops/automation/script/README-extra.md b/cm-mlops/automation/script/README-extra.md
index 9a4c8ab186..1a641bd785 100644
--- a/cm-mlops/automation/script/README-extra.md
+++ b/cm-mlops/automation/script/README-extra.md
@@ -511,7 +511,7 @@ to [detect or install/build Python interpreter](https://github.com/mlcommons/ck/
This script exposes a number of environment variables for a detected Python
in the [`postprocess` function](https://github.com/mlcommons/ck/blob/master/cm-mlops/script/get-python3/customize.py#L60):
-* `CM_PYTHON_BIN` - python3.10 or python.exe or any other name of a Python interpreteur on a given system
+* `CM_PYTHON_BIN` - python3.10 or python.exe or any other name of a Python interpreter on a given system
* `CM_PYTHON_BIN_PATH` - path to a detected or installed python
* `CM_PYTHON_BIN_WITH_PATH` - full path to a detected or installed python
* `LD_LIBRARY_PATH` - updated LD_LIBRARY_PATH to python
@@ -573,7 +573,7 @@ You can see the cached files as follows:
ls `cm find cache --tags=get,python`
```
-* _cm.json - CM meta description of this "cache" artifact with its uniqie ID, tags and other meta information
+* _cm.json - CM meta description of this "cache" artifact with its unique ID, tags and other meta information
* cm-cached-state.json - dictionary with the new environment variables and the new state dictionary
* tmp-env-all.sh - all environment variables used during CM script execution
* tmp-env.sh - only new environment variables produced after CM script execution (it can be used directly by external tools)
@@ -609,7 +609,7 @@ as shown in the next example.
### Assembling pipeline to compile and run image corner detection
-We can use automatically detected compiler from CM script to create simple and technology-netural compilation and execution pipelines
+We can use automatically detected compiler from CM script to create simple and technology-neutral compilation and execution pipelines
in CM scripts.
For example, we have implemented a simple [image corner detection CM script]( https://github.com/mlcommons/ck/tree/master/cm-mlops/script/app-image-corner-detection )
@@ -652,7 +652,7 @@ Note that this directory also contains the compiled tool "image-corner" that can
### Customizing sub-dependencies in a pipeline
When running a CM script with many sub-dependencies similar to above example,
-we may want to specify some version constaints on sub-dependencies such as LLVM.
+we may want to specify some version constraints on sub-dependencies such as LLVM.
One can use the key `"names"` in the "deps" list of any CM script meta description
to specify multiple names for a given dependency.
@@ -687,7 +687,7 @@ cm run script "install python-venv" --name={some name}
```
CM will create a virtual environment using default Python and save it in CM cache.
-It is possible to create a python virutal environment with any version on Linux and MacOS as follows:
+It is possible to create a python virtual environment with any version on Linux and MacOS as follows:
```bash
cm run script "install python-venv" --version=3.10.8 --name=mlperf
@@ -861,7 +861,7 @@ that's why we called our project "Collective Knowledge".
### Detecting, installing and caching system dependencies
-Many projects require installation of some system dependencies. Unfortuantely, the procedure
+Many projects require installation of some system dependencies. Unfortunately, the procedure
is different across different systems.
That's why we have developed two other CM script to unify and automate this process on any system.
diff --git a/cm-mlops/automation/script/README.md b/cm-mlops/automation/script/README.md
index db2c8ccf5c..937ae4bfbf 100644
--- a/cm-mlops/automation/script/README.md
+++ b/cm-mlops/automation/script/README.md
@@ -1,4 +1,4 @@
-*This README is automatically generated - don't edit! Use `README-extra.md` for extra notes!*
+*This README is automatically generated - don't edit! See [extra README](README-extra.md) for extra notes!*
### Automation actions
@@ -24,8 +24,8 @@
#### version
- * CM CLI: ```cm version script``` ([add flags (dict keys) from this API](https://github.com/mlcommons/ck/tree/master/cm-mlops/automation/script/module.py#L1656))
- * CM CLI with UID: ```cm version script,5b4e0237da074764``` ([add flags (dict keys) from this API](https://github.com/mlcommons/ck/tree/master/cm-mlops/automation/script/module.py#L1656))
+ * CM CLI: ```cm version script``` ([add flags (dict keys) from this API](https://github.com/mlcommons/ck/tree/master/cm-mlops/automation/script/module.py#L1697))
+ * CM CLI with UID: ```cm version script,5b4e0237da074764``` ([add flags (dict keys) from this API](https://github.com/mlcommons/ck/tree/master/cm-mlops/automation/script/module.py#L1697))
* CM Python API:
```python
import cmind
@@ -35,7 +35,7 @@
'automation':'script,5b4e0237da074764'
'out':'con'
```
- [add keys from this API](https://github.com/mlcommons/ck/tree/master/cm-mlops/automation/script/module.py#L1656)
+ [add keys from this API](https://github.com/mlcommons/ck/tree/master/cm-mlops/automation/script/module.py#L1697)
```python
})
if r['return']>0:
@@ -44,8 +44,8 @@
#### search
- * CM CLI: ```cm search script``` ([add flags (dict keys) from this API](https://github.com/mlcommons/ck/tree/master/cm-mlops/automation/script/module.py#L1684))
- * CM CLI with UID: ```cm search script,5b4e0237da074764``` ([add flags (dict keys) from this API](https://github.com/mlcommons/ck/tree/master/cm-mlops/automation/script/module.py#L1684))
+ * CM CLI: ```cm search script``` ([add flags (dict keys) from this API](https://github.com/mlcommons/ck/tree/master/cm-mlops/automation/script/module.py#L1725))
+ * CM CLI with UID: ```cm search script,5b4e0237da074764``` ([add flags (dict keys) from this API](https://github.com/mlcommons/ck/tree/master/cm-mlops/automation/script/module.py#L1725))
* CM Python API:
```python
import cmind
@@ -55,7 +55,7 @@
'automation':'script,5b4e0237da074764'
'out':'con'
```
- [add keys from this API](https://github.com/mlcommons/ck/tree/master/cm-mlops/automation/script/module.py#L1684)
+ [add keys from this API](https://github.com/mlcommons/ck/tree/master/cm-mlops/automation/script/module.py#L1725)
```python
})
if r['return']>0:
@@ -64,8 +64,8 @@
#### test
- * CM CLI: ```cm test script``` ([add flags (dict keys) from this API](https://github.com/mlcommons/ck/tree/master/cm-mlops/automation/script/module.py#L1792))
- * CM CLI with UID: ```cm test script,5b4e0237da074764``` ([add flags (dict keys) from this API](https://github.com/mlcommons/ck/tree/master/cm-mlops/automation/script/module.py#L1792))
+ * CM CLI: ```cm test script``` ([add flags (dict keys) from this API](https://github.com/mlcommons/ck/tree/master/cm-mlops/automation/script/module.py#L1835))
+ * CM CLI with UID: ```cm test script,5b4e0237da074764``` ([add flags (dict keys) from this API](https://github.com/mlcommons/ck/tree/master/cm-mlops/automation/script/module.py#L1835))
* CM Python API:
```python
import cmind
@@ -75,7 +75,7 @@
'automation':'script,5b4e0237da074764'
'out':'con'
```
- [add keys from this API](https://github.com/mlcommons/ck/tree/master/cm-mlops/automation/script/module.py#L1792)
+ [add keys from this API](https://github.com/mlcommons/ck/tree/master/cm-mlops/automation/script/module.py#L1835)
```python
})
if r['return']>0:
@@ -84,8 +84,8 @@
#### native_run
- * CM CLI: ```cm native_run script``` ([add flags (dict keys) from this API](https://github.com/mlcommons/ck/tree/master/cm-mlops/automation/script/module.py#L1858))
- * CM CLI with UID: ```cm native_run script,5b4e0237da074764``` ([add flags (dict keys) from this API](https://github.com/mlcommons/ck/tree/master/cm-mlops/automation/script/module.py#L1858))
+ * CM CLI: ```cm native_run script``` ([add flags (dict keys) from this API](https://github.com/mlcommons/ck/tree/master/cm-mlops/automation/script/module.py#L1901))
+ * CM CLI with UID: ```cm native_run script,5b4e0237da074764``` ([add flags (dict keys) from this API](https://github.com/mlcommons/ck/tree/master/cm-mlops/automation/script/module.py#L1901))
* CM Python API:
```python
import cmind
@@ -95,7 +95,7 @@
'automation':'script,5b4e0237da074764'
'out':'con'
```
- [add keys from this API](https://github.com/mlcommons/ck/tree/master/cm-mlops/automation/script/module.py#L1858)
+ [add keys from this API](https://github.com/mlcommons/ck/tree/master/cm-mlops/automation/script/module.py#L1901)
```python
})
if r['return']>0:
@@ -104,8 +104,8 @@
#### add
- * CM CLI: ```cm add script``` ([add flags (dict keys) from this API](https://github.com/mlcommons/ck/tree/master/cm-mlops/automation/script/module.py#L1931))
- * CM CLI with UID: ```cm add script,5b4e0237da074764``` ([add flags (dict keys) from this API](https://github.com/mlcommons/ck/tree/master/cm-mlops/automation/script/module.py#L1931))
+ * CM CLI: ```cm add script``` ([add flags (dict keys) from this API](https://github.com/mlcommons/ck/tree/master/cm-mlops/automation/script/module.py#L1974))
+ * CM CLI with UID: ```cm add script,5b4e0237da074764``` ([add flags (dict keys) from this API](https://github.com/mlcommons/ck/tree/master/cm-mlops/automation/script/module.py#L1974))
* CM Python API:
```python
import cmind
@@ -115,7 +115,7 @@
'automation':'script,5b4e0237da074764'
'out':'con'
```
- [add keys from this API](https://github.com/mlcommons/ck/tree/master/cm-mlops/automation/script/module.py#L1931)
+ [add keys from this API](https://github.com/mlcommons/ck/tree/master/cm-mlops/automation/script/module.py#L1974)
```python
})
if r['return']>0:
@@ -124,8 +124,8 @@
#### run_native_script
- * CM CLI: ```cm run_native_script script``` ([add flags (dict keys) from this API](https://github.com/mlcommons/ck/tree/master/cm-mlops/automation/script/module.py#L2399))
- * CM CLI with UID: ```cm run_native_script script,5b4e0237da074764``` ([add flags (dict keys) from this API](https://github.com/mlcommons/ck/tree/master/cm-mlops/automation/script/module.py#L2399))
+ * CM CLI: ```cm run_native_script script``` ([add flags (dict keys) from this API](https://github.com/mlcommons/ck/tree/master/cm-mlops/automation/script/module.py#L2463))
+ * CM CLI with UID: ```cm run_native_script script,5b4e0237da074764``` ([add flags (dict keys) from this API](https://github.com/mlcommons/ck/tree/master/cm-mlops/automation/script/module.py#L2463))
* CM Python API:
```python
import cmind
@@ -135,7 +135,7 @@
'automation':'script,5b4e0237da074764'
'out':'con'
```
- [add keys from this API](https://github.com/mlcommons/ck/tree/master/cm-mlops/automation/script/module.py#L2399)
+ [add keys from this API](https://github.com/mlcommons/ck/tree/master/cm-mlops/automation/script/module.py#L2463)
```python
})
if r['return']>0:
@@ -144,8 +144,8 @@
#### find_file_in_paths
- * CM CLI: ```cm find_file_in_paths script``` ([add flags (dict keys) from this API](https://github.com/mlcommons/ck/tree/master/cm-mlops/automation/script/module.py#L2440))
- * CM CLI with UID: ```cm find_file_in_paths script,5b4e0237da074764``` ([add flags (dict keys) from this API](https://github.com/mlcommons/ck/tree/master/cm-mlops/automation/script/module.py#L2440))
+ * CM CLI: ```cm find_file_in_paths script``` ([add flags (dict keys) from this API](https://github.com/mlcommons/ck/tree/master/cm-mlops/automation/script/module.py#L2504))
+ * CM CLI with UID: ```cm find_file_in_paths script,5b4e0237da074764``` ([add flags (dict keys) from this API](https://github.com/mlcommons/ck/tree/master/cm-mlops/automation/script/module.py#L2504))
* CM Python API:
```python
import cmind
@@ -155,7 +155,7 @@
'automation':'script,5b4e0237da074764'
'out':'con'
```
- [add keys from this API](https://github.com/mlcommons/ck/tree/master/cm-mlops/automation/script/module.py#L2440)
+ [add keys from this API](https://github.com/mlcommons/ck/tree/master/cm-mlops/automation/script/module.py#L2504)
```python
})
if r['return']>0:
@@ -164,8 +164,8 @@
#### detect_version_using_script
- * CM CLI: ```cm detect_version_using_script script``` ([add flags (dict keys) from this API](https://github.com/mlcommons/ck/tree/master/cm-mlops/automation/script/module.py#L2654))
- * CM CLI with UID: ```cm detect_version_using_script script,5b4e0237da074764``` ([add flags (dict keys) from this API](https://github.com/mlcommons/ck/tree/master/cm-mlops/automation/script/module.py#L2654))
+ * CM CLI: ```cm detect_version_using_script script``` ([add flags (dict keys) from this API](https://github.com/mlcommons/ck/tree/master/cm-mlops/automation/script/module.py#L2718))
+ * CM CLI with UID: ```cm detect_version_using_script script,5b4e0237da074764``` ([add flags (dict keys) from this API](https://github.com/mlcommons/ck/tree/master/cm-mlops/automation/script/module.py#L2718))
* CM Python API:
```python
import cmind
@@ -175,7 +175,7 @@
'automation':'script,5b4e0237da074764'
'out':'con'
```
- [add keys from this API](https://github.com/mlcommons/ck/tree/master/cm-mlops/automation/script/module.py#L2654)
+ [add keys from this API](https://github.com/mlcommons/ck/tree/master/cm-mlops/automation/script/module.py#L2718)
```python
})
if r['return']>0:
@@ -184,8 +184,8 @@
#### find_artifact
- * CM CLI: ```cm find_artifact script``` ([add flags (dict keys) from this API](https://github.com/mlcommons/ck/tree/master/cm-mlops/automation/script/module.py#L2727))
- * CM CLI with UID: ```cm find_artifact script,5b4e0237da074764``` ([add flags (dict keys) from this API](https://github.com/mlcommons/ck/tree/master/cm-mlops/automation/script/module.py#L2727))
+ * CM CLI: ```cm find_artifact script``` ([add flags (dict keys) from this API](https://github.com/mlcommons/ck/tree/master/cm-mlops/automation/script/module.py#L2791))
+ * CM CLI with UID: ```cm find_artifact script,5b4e0237da074764``` ([add flags (dict keys) from this API](https://github.com/mlcommons/ck/tree/master/cm-mlops/automation/script/module.py#L2791))
* CM Python API:
```python
import cmind
@@ -195,7 +195,7 @@
'automation':'script,5b4e0237da074764'
'out':'con'
```
- [add keys from this API](https://github.com/mlcommons/ck/tree/master/cm-mlops/automation/script/module.py#L2727)
+ [add keys from this API](https://github.com/mlcommons/ck/tree/master/cm-mlops/automation/script/module.py#L2791)
```python
})
if r['return']>0:
@@ -204,8 +204,8 @@
#### find_file_deep
- * CM CLI: ```cm find_file_deep script``` ([add flags (dict keys) from this API](https://github.com/mlcommons/ck/tree/master/cm-mlops/automation/script/module.py#L2885))
- * CM CLI with UID: ```cm find_file_deep script,5b4e0237da074764``` ([add flags (dict keys) from this API](https://github.com/mlcommons/ck/tree/master/cm-mlops/automation/script/module.py#L2885))
+ * CM CLI: ```cm find_file_deep script``` ([add flags (dict keys) from this API](https://github.com/mlcommons/ck/tree/master/cm-mlops/automation/script/module.py#L2949))
+ * CM CLI with UID: ```cm find_file_deep script,5b4e0237da074764``` ([add flags (dict keys) from this API](https://github.com/mlcommons/ck/tree/master/cm-mlops/automation/script/module.py#L2949))
* CM Python API:
```python
import cmind
@@ -215,7 +215,7 @@
'automation':'script,5b4e0237da074764'
'out':'con'
```
- [add keys from this API](https://github.com/mlcommons/ck/tree/master/cm-mlops/automation/script/module.py#L2885)
+ [add keys from this API](https://github.com/mlcommons/ck/tree/master/cm-mlops/automation/script/module.py#L2949)
```python
})
if r['return']>0:
@@ -224,8 +224,8 @@
#### find_file_back
- * CM CLI: ```cm find_file_back script``` ([add flags (dict keys) from this API](https://github.com/mlcommons/ck/tree/master/cm-mlops/automation/script/module.py#L2943))
- * CM CLI with UID: ```cm find_file_back script,5b4e0237da074764``` ([add flags (dict keys) from this API](https://github.com/mlcommons/ck/tree/master/cm-mlops/automation/script/module.py#L2943))
+ * CM CLI: ```cm find_file_back script``` ([add flags (dict keys) from this API](https://github.com/mlcommons/ck/tree/master/cm-mlops/automation/script/module.py#L3007))
+ * CM CLI with UID: ```cm find_file_back script,5b4e0237da074764``` ([add flags (dict keys) from this API](https://github.com/mlcommons/ck/tree/master/cm-mlops/automation/script/module.py#L3007))
* CM Python API:
```python
import cmind
@@ -235,7 +235,7 @@
'automation':'script,5b4e0237da074764'
'out':'con'
```
- [add keys from this API](https://github.com/mlcommons/ck/tree/master/cm-mlops/automation/script/module.py#L2943)
+ [add keys from this API](https://github.com/mlcommons/ck/tree/master/cm-mlops/automation/script/module.py#L3007)
```python
})
if r['return']>0:
@@ -244,8 +244,8 @@
#### parse_version
- * CM CLI: ```cm parse_version script``` ([add flags (dict keys) from this API](https://github.com/mlcommons/ck/tree/master/cm-mlops/automation/script/module.py#L2984))
- * CM CLI with UID: ```cm parse_version script,5b4e0237da074764``` ([add flags (dict keys) from this API](https://github.com/mlcommons/ck/tree/master/cm-mlops/automation/script/module.py#L2984))
+ * CM CLI: ```cm parse_version script``` ([add flags (dict keys) from this API](https://github.com/mlcommons/ck/tree/master/cm-mlops/automation/script/module.py#L3048))
+ * CM CLI with UID: ```cm parse_version script,5b4e0237da074764``` ([add flags (dict keys) from this API](https://github.com/mlcommons/ck/tree/master/cm-mlops/automation/script/module.py#L3048))
* CM Python API:
```python
import cmind
@@ -255,7 +255,7 @@
'automation':'script,5b4e0237da074764'
'out':'con'
```
- [add keys from this API](https://github.com/mlcommons/ck/tree/master/cm-mlops/automation/script/module.py#L2984)
+ [add keys from this API](https://github.com/mlcommons/ck/tree/master/cm-mlops/automation/script/module.py#L3048)
```python
})
if r['return']>0:
@@ -264,8 +264,8 @@
#### update_deps
- * CM CLI: ```cm update_deps script``` ([add flags (dict keys) from this API](https://github.com/mlcommons/ck/tree/master/cm-mlops/automation/script/module.py#L3038))
- * CM CLI with UID: ```cm update_deps script,5b4e0237da074764``` ([add flags (dict keys) from this API](https://github.com/mlcommons/ck/tree/master/cm-mlops/automation/script/module.py#L3038))
+ * CM CLI: ```cm update_deps script``` ([add flags (dict keys) from this API](https://github.com/mlcommons/ck/tree/master/cm-mlops/automation/script/module.py#L3102))
+ * CM CLI with UID: ```cm update_deps script,5b4e0237da074764``` ([add flags (dict keys) from this API](https://github.com/mlcommons/ck/tree/master/cm-mlops/automation/script/module.py#L3102))
* CM Python API:
```python
import cmind
@@ -275,7 +275,7 @@
'automation':'script,5b4e0237da074764'
'out':'con'
```
- [add keys from this API](https://github.com/mlcommons/ck/tree/master/cm-mlops/automation/script/module.py#L3038)
+ [add keys from this API](https://github.com/mlcommons/ck/tree/master/cm-mlops/automation/script/module.py#L3102)
```python
})
if r['return']>0:
@@ -284,8 +284,8 @@
#### get_default_path_list
- * CM CLI: ```cm get_default_path_list script``` ([add flags (dict keys) from this API](https://github.com/mlcommons/ck/tree/master/cm-mlops/automation/script/module.py#L3057))
- * CM CLI with UID: ```cm get_default_path_list script,5b4e0237da074764``` ([add flags (dict keys) from this API](https://github.com/mlcommons/ck/tree/master/cm-mlops/automation/script/module.py#L3057))
+ * CM CLI: ```cm get_default_path_list script``` ([add flags (dict keys) from this API](https://github.com/mlcommons/ck/tree/master/cm-mlops/automation/script/module.py#L3121))
+ * CM CLI with UID: ```cm get_default_path_list script,5b4e0237da074764``` ([add flags (dict keys) from this API](https://github.com/mlcommons/ck/tree/master/cm-mlops/automation/script/module.py#L3121))
* CM Python API:
```python
import cmind
@@ -295,7 +295,7 @@
'automation':'script,5b4e0237da074764'
'out':'con'
```
- [add keys from this API](https://github.com/mlcommons/ck/tree/master/cm-mlops/automation/script/module.py#L3057)
+ [add keys from this API](https://github.com/mlcommons/ck/tree/master/cm-mlops/automation/script/module.py#L3121)
```python
})
if r['return']>0:
@@ -304,8 +304,8 @@
#### doc
- * CM CLI: ```cm doc script``` ([add flags (dict keys) from this API](https://github.com/mlcommons/ck/tree/master/cm-mlops/automation/script/module.py#L3068))
- * CM CLI with UID: ```cm doc script,5b4e0237da074764``` ([add flags (dict keys) from this API](https://github.com/mlcommons/ck/tree/master/cm-mlops/automation/script/module.py#L3068))
+ * CM CLI: ```cm doc script``` ([add flags (dict keys) from this API](https://github.com/mlcommons/ck/tree/master/cm-mlops/automation/script/module.py#L3132))
+ * CM CLI with UID: ```cm doc script,5b4e0237da074764``` ([add flags (dict keys) from this API](https://github.com/mlcommons/ck/tree/master/cm-mlops/automation/script/module.py#L3132))
* CM Python API:
```python
import cmind
@@ -315,7 +315,47 @@
'automation':'script,5b4e0237da074764'
'out':'con'
```
- [add keys from this API](https://github.com/mlcommons/ck/tree/master/cm-mlops/automation/script/module.py#L3068)
+ [add keys from this API](https://github.com/mlcommons/ck/tree/master/cm-mlops/automation/script/module.py#L3132)
+ ```python
+ })
+ if r['return']>0:
+ print(r['error'])
+ ```
+
+#### dockerfile
+
+ * CM CLI: ```cm dockerfile script``` ([add flags (dict keys) from this API](https://github.com/mlcommons/ck/tree/master/cm-mlops/automation/script/module.py#L3161))
+ * CM CLI with UID: ```cm dockerfile script,5b4e0237da074764``` ([add flags (dict keys) from this API](https://github.com/mlcommons/ck/tree/master/cm-mlops/automation/script/module.py#L3161))
+ * CM Python API:
+ ```python
+ import cmind
+
+ r=cm.access({
+ 'action':'dockerfile'
+ 'automation':'script,5b4e0237da074764'
+ 'out':'con'
+ ```
+ [add keys from this API](https://github.com/mlcommons/ck/tree/master/cm-mlops/automation/script/module.py#L3161)
+ ```python
+ })
+ if r['return']>0:
+ print(r['error'])
+ ```
+
+#### docker
+
+ * CM CLI: ```cm docker script``` ([add flags (dict keys) from this API](https://github.com/mlcommons/ck/tree/master/cm-mlops/automation/script/module.py#L3189))
+ * CM CLI with UID: ```cm docker script,5b4e0237da074764``` ([add flags (dict keys) from this API](https://github.com/mlcommons/ck/tree/master/cm-mlops/automation/script/module.py#L3189))
+ * CM Python API:
+ ```python
+ import cmind
+
+ r=cm.access({
+ 'action':'docker'
+ 'automation':'script,5b4e0237da074764'
+ 'out':'con'
+ ```
+ [add keys from this API](https://github.com/mlcommons/ck/tree/master/cm-mlops/automation/script/module.py#L3189)
```python
})
if r['return']>0:
@@ -324,4 +364,4 @@
### Maintainers
-* [Open MLCommons taskforce on automation and reproducibility](https://github.com/mlcommons/ck/blob/master/docs/mlperf-education-workgroup.md)
\ No newline at end of file
+* [Open MLCommons taskforce on automation and reproducibility](https://cKnowledge.org/mlcommons-taskforce)
\ No newline at end of file
diff --git a/cm-mlops/automation/utils/README.md b/cm-mlops/automation/utils/README.md
index 34f46779a4..c5ee7326c5 100644
--- a/cm-mlops/automation/utils/README.md
+++ b/cm-mlops/automation/utils/README.md
@@ -244,4 +244,4 @@
### Maintainers
-* [Open MLCommons taskforce on automation and reproducibility](https://github.com/mlcommons/ck/blob/master/docs/mlperf-education-workgroup.md)
\ No newline at end of file
+* [Open MLCommons taskforce on automation and reproducibility](https://cKnowledge.org/mlcommons-taskforce)
\ No newline at end of file
diff --git a/cm-mlops/script/compile-program/README.md b/cm-mlops/script/compile-program/README.md
index ae157c8d26..b69b5f4cc0 100644
--- a/cm-mlops/script/compile-program/README.md
+++ b/cm-mlops/script/compile-program/README.md
@@ -123,9 +123,9 @@ ___
- CM script: [detect-cpu](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/detect-cpu)
* get,compiler
* CM names: `--adr.['compiler']...`
+ - CM script: [get-cl](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/get-cl)
- CM script: [get-gcc](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/get-gcc)
- CM script: [get-llvm](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/get-llvm)
- - CM script: [get-cl](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/get-cl)
* get,compiler-flags
- CM script: [get-compiler-flags](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/get-compiler-flags)
1. ***Run "preprocess" function from [customize.py](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/compile-program/customize.py)***
diff --git a/cm-mlops/script/download-and-extract/README.md b/cm-mlops/script/download-and-extract/README.md
index e2349fdd9d..0b5b9fd500 100644
--- a/cm-mlops/script/download-and-extract/README.md
+++ b/cm-mlops/script/download-and-extract/README.md
@@ -184,11 +184,13 @@ ___
* download,file
* `if (CM_DAE_DOWNLOAD_USING_TORRENT not in ['yes', 'True'])`
* CM names: `--adr.['download-script']...`
+ - CM script: [download-and-extract](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/download-and-extract)
- CM script: [download-file](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/download-file)
1. ***Run native script if exists***
1. ***Read "posthook_deps" on other CM scripts from [meta](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/download-and-extract/_cm.json)***
* extract,file
* `if (CM_DAE_EXTRACT_DOWNLOADED in ['yes', 'True'])`
+ - CM script: [download-and-extract](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/download-and-extract)
- CM script: [extract-file](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/extract-file)
1. ***Run "postrocess" function from [customize.py](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/download-and-extract/customize.py)***
1. Read "post_deps" on other CM scripts from [meta](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/download-and-extract/_cm.json)
diff --git a/cm-mlops/script/download-torrent/README.md b/cm-mlops/script/download-torrent/README.md
index db98d5831c..b528f96e73 100644
--- a/cm-mlops/script/download-torrent/README.md
+++ b/cm-mlops/script/download-torrent/README.md
@@ -12,6 +12,7 @@
* [ CM modular Docker container](#cm-modular-docker-container)
* [Customization](#customization)
* [ Variations](#variations)
+ * [ Script flags mapped to environment](#script-flags-mapped-to-environment)
* [ Default environment](#default-environment)
* [Script workflow, dependencies and native scripts](#script-workflow-dependencies-and-native-scripts)
* [Script output](#script-output)
@@ -49,11 +50,11 @@ ___
#### CM CLI
-1. `cm run script --tags=download,torrent,download-torrent[,variations] `
+1. `cm run script --tags=download,torrent,download-torrent[,variations] [--input_flags]`
-2. `cm run script "download torrent download-torrent[,variations]" `
+2. `cm run script "download torrent download-torrent[,variations]" [--input_flags]`
-3. `cm run script 69b752c5618e45bb `
+3. `cm run script 69b752c5618e45bb [--input_flags]`
* `variations` can be seen [here](#variations)
@@ -112,6 +113,21 @@ ___
+
+#### Script flags mapped to environment
+
+Click here to expand this section.
+
+* `--wait=value` → `CM_TORRENT_WAIT_UNTIL_COMPLETED=value`
+
+**Above CLI flags can be used in the Python CM API as follows:**
+
+```python
+r=cm.access({... , "wait":...}
+```
+
+
+
#### Default environment
@@ -119,6 +135,7 @@ ___
These keys can be updated via `--env.KEY=VALUE` or `env` dictionary in `@input.json` or using script flags.
+* CM_TORRENT_WAIT_UNTIL_COMPLETED: `no`
@@ -144,8 +161,11 @@ ___
### Script output
#### New environment keys (filter)
+* `<<>>`
+* `CM_TORRENT_DOWNLOADED_PATH`
#### New environment keys auto-detected from customize
+* `CM_TORRENT_DOWNLOADED_PATH`
___
### Maintainers
diff --git a/cm-mlops/script/generate-mlperf-tiny-submission/README.md b/cm-mlops/script/generate-mlperf-tiny-submission/README.md
index 8ea0aea7dd..70e31f377b 100644
--- a/cm-mlops/script/generate-mlperf-tiny-submission/README.md
+++ b/cm-mlops/script/generate-mlperf-tiny-submission/README.md
@@ -131,34 +131,10 @@ ___
1. ***Read "post_deps" on other CM scripts from [meta](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/generate-mlperf-tiny-submission/_cm.json)***
*
* `if (CM_MLPERF_RUN_STYLE == valid)`
- - CM script: [destroy-terraform](destroy-terraform)
- - CM script: [test](test)
- - CM script: [.](.)
- - CM script: [get-sut](https://github.com/arjunsuresh/cm-qaic/tree/master/script/get-sut)
- - CM script: [image-classification-qaic-loadgen](https://github.com/arjunsuresh/cm-qaic/tree/master/script/image-classification-qaic-loadgen)
- - CM script: [lib-qaic-api](https://github.com/arjunsuresh/cm-qaic/tree/master/script/lib-qaic-api)
- - CM script: [model-onnx-mlperf-packed-bert-for-qaic](https://github.com/arjunsuresh/cm-qaic/tree/master/script/model-onnx-mlperf-packed-bert-for-qaic)
- - CM script: [model-onnx-mlperf-retinanet-no-nms](https://github.com/arjunsuresh/cm-qaic/tree/master/script/model-onnx-mlperf-retinanet-no-nms)
- - CM script: [model-onnx-mlperf-ssd-resnet-no-nms](https://github.com/arjunsuresh/cm-qaic/tree/master/script/model-onnx-mlperf-ssd-resnet-no-nms)
- - CM script: [model-qaic-calibrate](https://github.com/arjunsuresh/cm-qaic/tree/master/script/model-qaic-calibrate)
- - CM script: [model-qaic-calibrate-bert](https://github.com/arjunsuresh/cm-qaic/tree/master/script/model-qaic-calibrate-bert)
- - CM script: [model-qaic-compile](https://github.com/arjunsuresh/cm-qaic/tree/master/script/model-qaic-compile)
- - CM script: [model-qaic-install-to-aedk](https://github.com/arjunsuresh/cm-qaic/tree/master/script/model-qaic-install-to-aedk)
- - CM script: [object-detection-qaic-loadgen](https://github.com/arjunsuresh/cm-qaic/tree/master/script/object-detection-qaic-loadgen)
- - CM script: [packed-bert-qaic-loadgen](https://github.com/arjunsuresh/cm-qaic/tree/master/script/packed-bert-qaic-loadgen)
- - CM script: [qaic-sys-config](https://github.com/arjunsuresh/cm-qaic/tree/master/script/qaic-sys-config)
- - CM script: [run-config](https://github.com/arjunsuresh/cm-qaic/tree/master/script/run-config)
- - CM script: [app-question-classification](https://github.com/GATEOverflow/topic-classification/tree/master/script/app-question-classification)
- - CM script: [get-dataset-question-topic-go](https://github.com/GATEOverflow/topic-classification/tree/master/script/get-dataset-question-topic-go)
- - CM script: [get-ml-model-question-classification](https://github.com/GATEOverflow/topic-classification/tree/master/script/get-ml-model-question-classification)
- - CM script: [get-preprocessed-dataset-question-topic-go/](https://github.com/GATEOverflow/topic-classification/tree/master/script/get-preprocessed-dataset-question-topic-go/)
- - CM script: [postprocess-question-classification](https://github.com/GATEOverflow/topic-classification/tree/master/script/postprocess-question-classification)
- - CM script: [train-ml-model-question-classification-rh](https://github.com/GATEOverflow/topic-classification/tree/master/script/train-ml-model-question-classification-rh)
- - CM script: [train-ml-model-question-classification-rt](https://github.com/GATEOverflow/topic-classification/tree/master/script/train-ml-model-question-classification-rt)
- CM script: [activate-python-venv](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/activate-python-venv)
- CM script: [add-custom-nvidia-system](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/add-custom-nvidia-system)
- - CM script: [app-image-classification-onnx-cpp](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/app-image-classification-onnx-cpp)
- CM script: [app-image-classification-onnx-py](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/app-image-classification-onnx-py)
+ - CM script: [app-image-classification-tf-onnx-cpp](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/app-image-classification-tf-onnx-cpp)
- CM script: [app-image-classification-torch-py](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/app-image-classification-torch-py)
- CM script: [app-image-classification-tvm-onnx-py](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/app-image-classification-tvm-onnx-py)
- CM script: [app-image-corner-detection](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/app-image-corner-detection)
@@ -171,22 +147,29 @@ ___
- CM script: [app-mlperf-inference-tflite-cpp](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/app-mlperf-inference-tflite-cpp)
- CM script: [benchmark-program](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/benchmark-program)
- CM script: [build-docker-image](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/build-docker-image)
+ - CM script: [build-dockerfile](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/build-dockerfile)
- CM script: [build-mlperf-inference-server-nvidia](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/build-mlperf-inference-server-nvidia)
- CM script: [compile-program](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/compile-program)
- CM script: [convert-ml-model-huggingface-to-onnx](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/convert-ml-model-huggingface-to-onnx)
- CM script: [destroy-terraform](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/destroy-terraform)
- CM script: [detect-cpu](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/detect-cpu)
- CM script: [detect-os](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/detect-os)
+ - CM script: [download-and-extract](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/download-and-extract)
+ - CM script: [download-torrent](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/download-torrent)
- CM script: [flash-tinyml-binary](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/flash-tinyml-binary)
- CM script: [generate-mlperf-inference-submission](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/generate-mlperf-inference-submission)
- CM script: [generate-mlperf-inference-user-conf](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/generate-mlperf-inference-user-conf)
- CM script: [generate-mlperf-tiny-submission](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/generate-mlperf-tiny-submission)
- CM script: [generate-nvidia-engine](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/generate-nvidia-engine)
- CM script: [get-android-sdk](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/get-android-sdk)
+ - CM script: [get-aocl](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/get-aocl)
- CM script: [get-aws-cli](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/get-aws-cli)
- CM script: [get-bazel](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/get-bazel)
+ - CM script: [get-blis](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/get-blis)
+ - CM script: [get-brew](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/get-brew)
- CM script: [get-ck](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/get-ck)
- CM script: [get-ck-repo-mlops](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/get-ck-repo-mlops)
+ - CM script: [get-cl](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/get-cl)
- CM script: [get-cmake](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/get-cmake)
- CM script: [get-cmsis_5](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/get-cmsis_5)
- CM script: [get-compiler-flags](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/get-compiler-flags)
@@ -196,6 +179,7 @@ ___
- CM script: [get-dataset-criteo](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/get-dataset-criteo)
- CM script: [get-dataset-imagenet-aux](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/get-dataset-imagenet-aux)
- CM script: [get-dataset-imagenet-helper](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/get-dataset-imagenet-helper)
+ - CM script: [get-dataset-imagenet-train](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/get-dataset-imagenet-train)
- CM script: [get-dataset-imagenet-val](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/get-dataset-imagenet-val)
- CM script: [get-dataset-kits19](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/get-dataset-kits19)
- CM script: [get-dataset-librispeech](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/get-dataset-librispeech)
@@ -203,12 +187,14 @@ ___
- CM script: [get-dataset-openimages-annotations](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/get-dataset-openimages-annotations)
- CM script: [get-dataset-squad](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/get-dataset-squad)
- CM script: [get-dataset-squad-vocab](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/get-dataset-squad-vocab)
+ - CM script: [get-dlrm](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/get-dlrm)
- CM script: [get-gcc](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/get-gcc)
- CM script: [get-generic-python-lib](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/get-generic-python-lib)
- CM script: [get-generic-sys-util](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/get-generic-sys-util)
- CM script: [get-git-repo](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/get-git-repo)
- CM script: [get-github-cli](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/get-github-cli)
- CM script: [get-go](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/get-go)
+ - CM script: [get-google-test](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/get-google-test)
- CM script: [get-ipol-src](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/get-ipol-src)
- CM script: [get-java](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/get-java)
- CM script: [get-javac](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/get-javac)
@@ -219,6 +205,7 @@ ___
- CM script: [get-ml-model-3d-unet-kits19](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/get-ml-model-3d-unet-kits19)
- CM script: [get-ml-model-bert-base-squad](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/get-ml-model-bert-base-squad)
- CM script: [get-ml-model-bert-large-squad](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/get-ml-model-bert-large-squad)
+ - CM script: [get-ml-model-dlrm-terabyte](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/get-ml-model-dlrm-terabyte)
- CM script: [get-ml-model-efficientnet-lite](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/get-ml-model-efficientnet-lite)
- CM script: [get-ml-model-huggingface-zoo](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/get-ml-model-huggingface-zoo)
- CM script: [get-ml-model-mobilenet](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/get-ml-model-mobilenet)
@@ -233,13 +220,7 @@ ___
- CM script: [get-mlperf-inference-nvidia-common-code](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/get-mlperf-inference-nvidia-common-code)
- CM script: [get-mlperf-inference-results](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/get-mlperf-inference-results)
- CM script: [get-mlperf-inference-src](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/get-mlperf-inference-src)
- - CM script: [build-dockerfile](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/build-dockerfile)
- - CM script: [get-cl](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/get-cl)
- - CM script: [get-dlrm](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/get-dlrm)
- - CM script: [get-ml-model-dlrm-terabyte](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/get-ml-model-dlrm-terabyte)
- CM script: [get-mlperf-inference-sut-configs](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/get-mlperf-inference-sut-configs)
- - CM script: [get-terraform](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/get-terraform)
- - CM script: [install-python-src](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/install-python-src)
- CM script: [get-mlperf-inference-sut-description](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/get-mlperf-inference-sut-description)
- CM script: [get-mlperf-power-dev](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/get-mlperf-power-dev)
- CM script: [get-mlperf-training-src](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/get-mlperf-training-src)
@@ -252,15 +233,21 @@ ___
- CM script: [get-preprocessed-dataset-librispeech](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/get-preprocessed-dataset-librispeech)
- CM script: [get-preprocessed-dataset-openimages](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/get-preprocessed-dataset-openimages)
- CM script: [get-python3](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/get-python3)
+ - CM script: [get-qaic-compute-sdk](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/get-qaic-compute-sdk)
+ - CM script: [get-qaic-software-kit](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/get-qaic-software-kit)
+ - CM script: [get-rclone](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/get-rclone)
- CM script: [get-spec-ptd](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/get-spec-ptd)
- CM script: [get-sys-utils-cm](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/get-sys-utils-cm)
- CM script: [get-sys-utils-min](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/get-sys-utils-min)
- CM script: [get-tensorrt](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/get-tensorrt)
+ - CM script: [get-terraform](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/get-terraform)
- CM script: [get-tvm](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/get-tvm)
+ - CM script: [get-zendnn](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/get-zendnn)
- CM script: [get-zephyr](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/get-zephyr)
- CM script: [get-zephyr-sdk](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/get-zephyr-sdk)
- CM script: [gui](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/gui)
- CM script: [import-mlperf-inference-to-experiment](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/import-mlperf-inference-to-experiment)
+ - CM script: [import-mlperf-tiny-to-experiment](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/import-mlperf-tiny-to-experiment)
- CM script: [install-aws-cli](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/install-aws-cli)
- CM script: [install-bazel](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/install-bazel)
- CM script: [install-cmake-prebuilt](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/install-cmake-prebuilt)
@@ -272,6 +259,7 @@ ___
- CM script: [install-llvm-prebuilt](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/install-llvm-prebuilt)
- CM script: [install-llvm-src](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/install-llvm-src)
- CM script: [install-openssl](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/install-openssl)
+ - CM script: [install-python-src](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/install-python-src)
- CM script: [install-python-venv](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/install-python-venv)
- CM script: [install-tensorflow-for-c](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/install-tensorflow-for-c)
- CM script: [install-tensorflow-from-src](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/install-tensorflow-from-src)
@@ -298,9 +286,26 @@ ___
- CM script: [run-terraform](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/run-terraform)
- CM script: [set-echo-off-win](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/set-echo-off-win)
- CM script: [tar-my-folder](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/tar-my-folder)
+ - CM script: [test-mlperf-inference-retinanet-win](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/test-mlperf-inference-retinanet-win)
- CM script: [test-set-sys-user-cm](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/test-set-sys-user-cm)
- CM script: [truncate-mlperf-inference-accuracy-log](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/truncate-mlperf-inference-accuracy-log)
- CM script: [wrapper-reproduce-octoml-tinyml-submission](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/wrapper-reproduce-octoml-tinyml-submission)
+ - CM script: [test-script1](https://github.com/gfursin/cm-tests/tree/master/script/test-script1)
+ - CM script: [test-script2](https://github.com/gfursin/cm-tests/tree/master/script/test-script2)
+ - CM script: [test-script3](https://github.com/gfursin/cm-tests/tree/master/script/test-script3)
+ - CM script: [test-script4](https://github.com/gfursin/cm-tests/tree/master/script/test-script4)
+ - CM script: [test-script5](https://github.com/gfursin/cm-tests/tree/master/script/test-script5)
+ - CM script: [app-generate-image-dalle-mini-jax-py](https://github.com/cknowledge/tests/tree/master/cm/script/app-generate-image-dalle-mini-jax-py)
+ - CM script: [app-generate-image-stable-diffusion2-pytorch-cuda-py](https://github.com/cknowledge/tests/tree/master/cm/script/app-generate-image-stable-diffusion2-pytorch-cuda-py)
+ - CM script: [app-image-classification-onnx-py-ck](https://github.com/cknowledge/tests/tree/master/cm/script/app-image-classification-onnx-py-ck)
+ - CM script: [app-image-corner-detection-old](https://github.com/cknowledge/tests/tree/master/cm/script/app-image-corner-detection-old)
+ - CM script: [app-ipol-demo](https://github.com/cknowledge/tests/tree/master/cm/script/app-ipol-demo)
+ - CM script: [app-stable-diffusion-pytorch-cuda-py](https://github.com/cknowledge/tests/tree/master/cm/script/app-stable-diffusion-pytorch-cuda-py)
+ - CM script: [not-needed--get-android-cmdline-tools](https://github.com/cknowledge/tests/tree/master/cm/script/not-needed--get-android-cmdline-tools)
+ - CM script: [not-needed--install-android-cmdline-tools](https://github.com/cknowledge/tests/tree/master/cm/script/not-needed--install-android-cmdline-tools)
+ - CM script: [process-mlperf-inference-results](https://github.com/mlcommons/cm_inference_results/tree/master/script/process-mlperf-inference-results)
+ - CM script: [download-file](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/download-file)
+ - CM script: [extract-file](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/extract-file)
___
diff --git a/cm-mlops/script/get-blis/README.md b/cm-mlops/script/get-blis/README.md
index e6d6d8612c..0dbd95d2cd 100644
--- a/cm-mlops/script/get-blis/README.md
+++ b/cm-mlops/script/get-blis/README.md
@@ -34,8 +34,8 @@ See [more info](README-extra.md).
* CM GitHub repository: *[mlcommons@ck](https://github.com/mlcommons/ck/tree/master/cm-mlops)*
* GitHub directory for this script: *[GitHub](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/get-blis)*
* CM meta description for this script: *[_cm.json](_cm.json)*
-* CM "database" tags to find this script: *get,blis*
-* Output cached?: *False*
+* CM "database" tags to find this script: *get,lib,blis*
+* Output cached?: *True*
___
### Usage
@@ -53,9 +53,9 @@ ___
#### CM CLI
-1. `cm run script --tags=get,blis[,variations] `
+1. `cm run script --tags=get,lib,blis[,variations] `
-2. `cm run script "get blis[,variations]" `
+2. `cm run script "get lib blis[,variations]" `
3. `cm run script ea6e1cf75242456c `
@@ -74,7 +74,7 @@ import cmind
r = cmind.access({'action':'run'
'automation':'script',
- 'tags':'get,blis'
+ 'tags':'get,lib,blis'
'out':'con',
...
(other input keys for this script)
@@ -91,9 +91,9 @@ if r['return']>0:
#### CM GUI
-```cm run script --tags=gui --script="get,blis"```
+```cm run script --tags=gui --script="get,lib,blis"```
-Use this [online GUI](https://cKnowledge.org/cm-gui/?tags=get,blis) to generate CM CMD.
+Use this [online GUI](https://cKnowledge.org/cm-gui/?tags=get,lib,blis) to generate CM CMD.
#### CM modular Docker container
@@ -142,7 +142,7 @@ ___
Click here to expand this section.
1. ***Read "deps" on other CM scripts from [meta](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/get-blis/_cm.json)***
- * get,git,_repo.https://github.com/flame/blis.git
+ * get,git
* CM names: `--adr.['blis-source-repo']...`
- CM script: [get-git-repo](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/get-git-repo)
* detect,cpu
@@ -163,11 +163,15 @@ ___
* `+LD_LIBRARY_PATH`
* `CM_BLIS_INC`
+* `CM_BLIS_INSTALL_PATH`
* `CM_BLIS_LIB`
-* `CM_LIBFLAME_SRC_PATH`
+* `CM_BLIS_SRC_PATH`
#### New environment keys auto-detected from customize
-* `CM_LIBFLAME_SRC_PATH`
+* `CM_BLIS_INC`
+* `CM_BLIS_INSTALL_PATH`
+* `CM_BLIS_LIB`
+* `CM_BLIS_SRC_PATH`
___
### Maintainers
diff --git a/cm-mlops/script/get-brew/README.md b/cm-mlops/script/get-brew/README.md
index c855ce1cfe..507509bc51 100644
--- a/cm-mlops/script/get-brew/README.md
+++ b/cm-mlops/script/get-brew/README.md
@@ -20,13 +20,10 @@
-*Note that this README is automatically generated - don't edit! See [more info](README-extra.md).*
+*Note that this README is automatically generated - don't edit! Use `README-extra.md` to add more info.*
### Description
-
-See [more info](README-extra.md).
-
#### Information
* CM GitHub repository: *[mlcommons@ck](https://github.com/mlcommons/ck/tree/master/cm-mlops)*
diff --git a/cm-mlops/script/get-compiler-flags/README.md b/cm-mlops/script/get-compiler-flags/README.md
index 610297e730..2797c0cb80 100644
--- a/cm-mlops/script/get-compiler-flags/README.md
+++ b/cm-mlops/script/get-compiler-flags/README.md
@@ -120,9 +120,9 @@ ___
* get,compiler
* `if (CM_C_COMPILER_BIN != on)`
* CM names: `--adr.['compiler']...`
+ - CM script: [get-cl](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/get-cl)
- CM script: [get-gcc](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/get-gcc)
- CM script: [get-llvm](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/get-llvm)
- - CM script: [get-cl](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/get-cl)
1. ***Run "preprocess" function from [customize.py](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/get-compiler-flags/customize.py)***
1. Read "prehook_deps" on other CM scripts from [meta](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/get-compiler-flags/_cm.json)
1. ***Run native script if exists***
diff --git a/cm-mlops/script/get-dataset-imagenet-val/README.md b/cm-mlops/script/get-dataset-imagenet-val/README.md
index 96ebff8355..0f3f1dc38f 100644
--- a/cm-mlops/script/get-dataset-imagenet-val/README.md
+++ b/cm-mlops/script/get-dataset-imagenet-val/README.md
@@ -12,6 +12,7 @@
* [ CM modular Docker container](#cm-modular-docker-container)
* [Customization](#customization)
* [ Variations](#variations)
+ * [ Script flags mapped to environment](#script-flags-mapped-to-environment)
* [ Default environment](#default-environment)
* [Script workflow, dependencies and native scripts](#script-workflow-dependencies-and-native-scripts)
* [Script output](#script-output)
@@ -53,11 +54,11 @@ ___
#### CM CLI
-1. `cm run script --tags=get,val,validation,dataset,imagenet,ILSVRC,image-classification,original[,variations] `
+1. `cm run script --tags=get,val,validation,dataset,imagenet,ILSVRC,image-classification,original[,variations] [--input_flags]`
-2. `cm run script "get val validation dataset imagenet ILSVRC image-classification original[,variations]" `
+2. `cm run script "get val validation dataset imagenet ILSVRC image-classification original[,variations]" [--input_flags]`
-3. `cm run script 7afd58d287fe4f11 `
+3. `cm run script 7afd58d287fe4f11 [--input_flags]`
* `variations` can be seen [here](#variations)
@@ -113,13 +114,6 @@ ___
- Workflow:
* `_2012-full`
- Workflow:
- * `_full`
- - Environment variables:
- - *CM_DATASET_SIZE*: `50000`
- - *CM_IMAGENET_FULL*: `yes`
- - *CM_DAE_DOWNLOADED_FILENAME*: `ILSVRC2012_img_val.tar`
- - *CM_DAE_DOWNLOADED_CHECKSUM*: `29b22e2961454d5413ddabcf34fc5622`
- - Workflow:
@@ -128,6 +122,13 @@ ___
Click here to expand this section.
+ * `_full`
+ - Environment variables:
+ - *CM_DATASET_SIZE*: `50000`
+ - *CM_IMAGENET_FULL*: `yes`
+ - *CM_DAE_FILENAME*: `ILSVRC2012_img_val.tar`
+ - *CM_DAE_DOWNLOADED_CHECKSUM*: `29b22e2961454d5413ddabcf34fc5622`
+ - Workflow:
* `_size.#`
- Environment variables:
- *CM_DATASET_SIZE*: `#`
@@ -135,6 +136,7 @@ ___
* **`_size.500`** (default)
- Environment variables:
- *CM_DATASET_SIZE*: `500`
+ - *CM_DAE_FILENAME*: `ILSVRC2012_img_val_500.tar`
- *CM_DAE_URL*: `https://www.dropbox.com/s/57s11df6pts3z69/ILSVRC2012_img_val_500.tar`
- Workflow:
@@ -156,6 +158,21 @@ ___
#### Default variations
`_2012,_size.500`
+
+#### Script flags mapped to environment
+
+Click here to expand this section.
+
+* `--torrent=value` → `CM_DATASET_IMAGENET_VAL_TORRENT_PATH=value`
+
+**Above CLI flags can be used in the Python CM API as follows:**
+
+```python
+r=cm.access({... , "torrent":...}
+```
+
+
+
#### Default environment
@@ -177,13 +194,13 @@ ___
- CM script: [detect-os](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/detect-os)
1. ***Run "preprocess" function from [customize.py](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/get-dataset-imagenet-val/customize.py)***
1. ***Read "prehook_deps" on other CM scripts from [meta](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/get-dataset-imagenet-val/_cm.json)***
- * download,torrent
- * `if (CM_DATASET_IMAGENET_VAL_REQUIRE_TORRENT == yes)`
- * CM names: `--adr.['download-torrent']...`
- - CM script: [download-torrent](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/download-torrent)
- * download,extract,file,_extract
- * `if (CM_DATASET_IMAGENET_VAL_REQUIRE_DAE == yes)`
+ * download-and-extract,file,_extract
+ * `if (CM_DATASET_IMAGENET_VAL_REQUIRE_DAE in ['yes', 'True'])`
+ - CM script: [download-and-extract](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/download-and-extract)
+ * file,extract
+ * `if (CM_DAE_ONLY_EXTRACT in ['yes', 'True'])`
- CM script: [download-and-extract](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/download-and-extract)
+ - CM script: [extract-file](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/extract-file)
1. ***Run native script if exists***
* [run.bat](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/get-dataset-imagenet-val/run.bat)
1. Read "posthook_deps" on other CM scripts from [meta](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/get-dataset-imagenet-val/_cm.json)
diff --git a/cm-mlops/script/get-mlperf-inference-sut-description/README.md b/cm-mlops/script/get-mlperf-inference-sut-description/README.md
index 5f25c7ecaa..6351f205b9 100644
--- a/cm-mlops/script/get-mlperf-inference-sut-description/README.md
+++ b/cm-mlops/script/get-mlperf-inference-sut-description/README.md
@@ -142,9 +142,9 @@ ___
- CM script: [get-python3](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/get-python3)
* get,compiler
* CM names: `--adr.['compiler']...`
+ - CM script: [get-cl](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/get-cl)
- CM script: [get-gcc](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/get-gcc)
- CM script: [get-llvm](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/get-llvm)
- - CM script: [get-cl](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/get-cl)
* get,cuda-devices
* `if (CM_MLPERF_DEVICE in ['gpu', 'cuda'])`
- CM script: [get-cuda-devices](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/get-cuda-devices)
diff --git a/cm-mlops/script/get-qaic-software-kit/README.md b/cm-mlops/script/get-qaic-software-kit/README.md
index 08d88f2eb0..acc29f2c54 100644
--- a/cm-mlops/script/get-qaic-software-kit/README.md
+++ b/cm-mlops/script/get-qaic-software-kit/README.md
@@ -122,9 +122,9 @@ ___
- CM script: [get-cmake](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/get-cmake)
* get,compiler
* CM names: `--adr.['compiler']...`
- - CM script: [get-llvm](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/get-llvm)
- - CM script: [get-gcc](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/get-gcc)
- CM script: [get-cl](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/get-cl)
+ - CM script: [get-gcc](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/get-gcc)
+ - CM script: [get-llvm](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/get-llvm)
* get,generic,sys-util,_libudev-dev
- CM script: [get-generic-sys-util](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/get-generic-sys-util)
* get,generic,sys-util,_libpci-dev
diff --git a/cm-mlops/script/get-zendnn/README.md b/cm-mlops/script/get-zendnn/README.md
index 48d2e3fa1f..841bf38abb 100644
--- a/cm-mlops/script/get-zendnn/README.md
+++ b/cm-mlops/script/get-zendnn/README.md
@@ -20,13 +20,10 @@
-*Note that this README is automatically generated - don't edit! See [more info](README-extra.md).*
+*Note that this README is automatically generated - don't edit! Use `README-extra.md` to add more info.*
### Description
-
-See [more info](README-extra.md).
-
#### Information
* CM GitHub repository: *[mlcommons@ck](https://github.com/mlcommons/ck/tree/master/cm-mlops)*
diff --git a/cm-mlops/script/import-mlperf-inference-to-experiment/README.md b/cm-mlops/script/import-mlperf-inference-to-experiment/README.md
index dd971ff720..e4268410f5 100644
--- a/cm-mlops/script/import-mlperf-inference-to-experiment/README.md
+++ b/cm-mlops/script/import-mlperf-inference-to-experiment/README.md
@@ -11,6 +11,7 @@
* [ CM GUI](#cm-gui)
* [ CM modular Docker container](#cm-modular-docker-container)
* [Customization](#customization)
+ * [ Script flags mapped to environment](#script-flags-mapped-to-environment)
* [ Default environment](#default-environment)
* [Script workflow, dependencies and native scripts](#script-workflow-dependencies-and-native-scripts)
* [Script output](#script-output)
@@ -20,10 +21,13 @@
-*Note that this README is automatically generated - don't edit! Use `README-extra.md` to add more info.*
+*Note that this README is automatically generated - don't edit! See [more info](README-extra.md).*
### Description
+
+See [more info](README-extra.md).
+
#### Information
* Category: *Modular MLPerf benchmarks.*
@@ -49,11 +53,11 @@ ___
#### CM CLI
-1. `cm run script --tags=import,mlperf,inference,mlperf-inference,experiment,2experiment,to-experiment `
+1. `cm run script --tags=import,mlperf,inference,mlperf-inference,experiment,2experiment,to-experiment [--input_flags]`
-2. `cm run script "import mlperf inference mlperf-inference experiment 2experiment to-experiment" `
+2. `cm run script "import mlperf inference mlperf-inference experiment 2experiment to-experiment" [--input_flags]`
-3. `cm run script 72099fa962ea499c `
+3. `cm run script 72099fa962ea499c [--input_flags]`
* `variations` can be seen [here](#variations)
@@ -98,6 +102,21 @@ Use this [online GUI](https://cKnowledge.org/cm-gui/?tags=import,mlperf,inferenc
___
### Customization
+
+#### Script flags mapped to environment
+
+Click here to expand this section.
+
+* `--target_repo=value` → `CM_IMPORT_MLPERF_INFERENCE_TARGET_REPO=value`
+
+**Above CLI flags can be used in the Python CM API as follows:**
+
+```python
+r=cm.access({... , "target_repo":...}
+```
+
+
+
#### Default environment
diff --git a/cm-mlops/script/import-mlperf-tiny-to-experiment/README.md b/cm-mlops/script/import-mlperf-tiny-to-experiment/README.md
index dd971ff720..4153f9f822 100644
--- a/cm-mlops/script/import-mlperf-tiny-to-experiment/README.md
+++ b/cm-mlops/script/import-mlperf-tiny-to-experiment/README.md
@@ -11,6 +11,7 @@
* [ CM GUI](#cm-gui)
* [ CM modular Docker container](#cm-modular-docker-container)
* [Customization](#customization)
+ * [ Script flags mapped to environment](#script-flags-mapped-to-environment)
* [ Default environment](#default-environment)
* [Script workflow, dependencies and native scripts](#script-workflow-dependencies-and-native-scripts)
* [Script output](#script-output)
@@ -20,17 +21,20 @@
-*Note that this README is automatically generated - don't edit! Use `README-extra.md` to add more info.*
+*Note that this README is automatically generated - don't edit! See [more info](README-extra.md).*
### Description
+
+See [more info](README-extra.md).
+
#### Information
* Category: *Modular MLPerf benchmarks.*
* CM GitHub repository: *[mlcommons@ck](https://github.com/mlcommons/ck/tree/master/cm-mlops)*
-* GitHub directory for this script: *[GitHub](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/import-mlperf-inference-to-experiment)*
+* GitHub directory for this script: *[GitHub](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/import-mlperf-tiny-to-experiment)*
* CM meta description for this script: *[_cm.yaml](_cm.yaml)*
-* CM "database" tags to find this script: *import,mlperf,inference,mlperf-inference,experiment,2experiment,to-experiment*
+* CM "database" tags to find this script: *import,mlperf,tiny,mlperf-tiny,experiment,2experiment,to-experiment*
* Output cached?: *False*
___
### Usage
@@ -49,11 +53,11 @@ ___
#### CM CLI
-1. `cm run script --tags=import,mlperf,inference,mlperf-inference,experiment,2experiment,to-experiment `
+1. `cm run script --tags=import,mlperf,tiny,mlperf-tiny,experiment,2experiment,to-experiment [--input_flags]`
-2. `cm run script "import mlperf inference mlperf-inference experiment 2experiment to-experiment" `
+2. `cm run script "import mlperf tiny mlperf-tiny experiment 2experiment to-experiment" [--input_flags]`
-3. `cm run script 72099fa962ea499c `
+3. `cm run script 83e3efd7611f469b [--input_flags]`
* `variations` can be seen [here](#variations)
@@ -70,7 +74,7 @@ import cmind
r = cmind.access({'action':'run'
'automation':'script',
- 'tags':'import,mlperf,inference,mlperf-inference,experiment,2experiment,to-experiment'
+ 'tags':'import,mlperf,tiny,mlperf-tiny,experiment,2experiment,to-experiment'
'out':'con',
...
(other input keys for this script)
@@ -87,9 +91,9 @@ if r['return']>0:
#### CM GUI
-```cm run script --tags=gui --script="import,mlperf,inference,mlperf-inference,experiment,2experiment,to-experiment"```
+```cm run script --tags=gui --script="import,mlperf,tiny,mlperf-tiny,experiment,2experiment,to-experiment"```
-Use this [online GUI](https://cKnowledge.org/cm-gui/?tags=import,mlperf,inference,mlperf-inference,experiment,2experiment,to-experiment) to generate CM CMD.
+Use this [online GUI](https://cKnowledge.org/cm-gui/?tags=import,mlperf,tiny,mlperf-tiny,experiment,2experiment,to-experiment) to generate CM CMD.
#### CM modular Docker container
@@ -98,6 +102,21 @@ Use this [online GUI](https://cKnowledge.org/cm-gui/?tags=import,mlperf,inferenc
___
### Customization
+
+#### Script flags mapped to environment
+
+Click here to expand this section.
+
+* `--target_repo=value` → `CM_IMPORT_TINYMLPERF_TARGET_REPO=value`
+
+**Above CLI flags can be used in the Python CM API as follows:**
+
+```python
+r=cm.access({... , "target_repo":...}
+```
+
+
+
#### Default environment
@@ -114,17 +133,17 @@ ___
Click here to expand this section.
- 1. ***Read "deps" on other CM scripts from [meta](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/import-mlperf-inference-to-experiment/_cm.yaml)***
+ 1. ***Read "deps" on other CM scripts from [meta](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/import-mlperf-tiny-to-experiment/_cm.yaml)***
* detect,os
- CM script: [detect-os](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/detect-os)
* get,sys-utils-cm
- CM script: [get-sys-utils-cm](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/get-sys-utils-cm)
- 1. ***Run "preprocess" function from [customize.py](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/import-mlperf-inference-to-experiment/customize.py)***
- 1. Read "prehook_deps" on other CM scripts from [meta](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/import-mlperf-inference-to-experiment/_cm.yaml)
+ 1. ***Run "preprocess" function from [customize.py](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/import-mlperf-tiny-to-experiment/customize.py)***
+ 1. Read "prehook_deps" on other CM scripts from [meta](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/import-mlperf-tiny-to-experiment/_cm.yaml)
1. ***Run native script if exists***
- 1. Read "posthook_deps" on other CM scripts from [meta](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/import-mlperf-inference-to-experiment/_cm.yaml)
+ 1. Read "posthook_deps" on other CM scripts from [meta](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/import-mlperf-tiny-to-experiment/_cm.yaml)
1. Run "postrocess" function from customize.py
- 1. Read "post_deps" on other CM scripts from [meta](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/import-mlperf-inference-to-experiment/_cm.yaml)
+ 1. Read "post_deps" on other CM scripts from [meta](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/import-mlperf-tiny-to-experiment/_cm.yaml)
___
diff --git a/cm-mlops/script/reproduce-mlperf-inference-nvidia/README.md b/cm-mlops/script/reproduce-mlperf-inference-nvidia/README.md
index 2ebfba81c8..82dbc606b4 100644
--- a/cm-mlops/script/reproduce-mlperf-inference-nvidia/README.md
+++ b/cm-mlops/script/reproduce-mlperf-inference-nvidia/README.md
@@ -51,7 +51,24 @@ cm run script --tags=get,tensorrt --input=
On other systems you can do a package manager install and then CM should pick up the installation automatically during the workflow run.
-Nvidia run configuration values for each model-sceraio for known systems are stored in `__init__.py` files under configs directory. For custom systems these are stored under `custom.py` files. When custom config files are generated they override the default config values with empty ones (not desirable). So, you'll probably need to open the custom config file and comment out the overrides. Typically `gpu_batch_size` and `offline_expected_qps` are enough for an offline scenario run on a typical single GPU system.
+## Managing the configuration files
+
+
+```
+cm run script --tags=build,nvidia,inference,server
+```
+Once the below command is done, inference server will be build and your system will be automatically detected by Nvidia scripts.
+
+Nividia code location is output by the below command
+```
+cd `cm find cache --tags=inference,results,mlperf,_custom`
+```
+
+Further cd into inference_results_v<>/closed/NVIDIA
+
+Nvidia run configuration values for each model-sceraio for known systems are stored in `__init__.py` files under configs directory. For custom systems (ones which are different from the ones used by Nvidia for submission) these are stored under `custom.py` files.
+
+**Important** When custom config files are generated they override the default config values with empty ones (not desirable). So, you'll probably need to open the custom config file and comment out the overrides. Typically `gpu_batch_size` and `offline_expected_qps` are enough for an offline scenario run on a typical single GPU system.
@@ -390,6 +407,8 @@ ___
* get,generic-python-lib,_torchvision
* CM names: `--adr.['torchvision']...`
- CM script: [get-generic-python-lib](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/get-generic-python-lib)
+ * get,generic-python-lib,_torch
+ - CM script: [get-generic-python-lib](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/get-generic-python-lib)
* get,generic-python-lib,_nvidia-apex
- CM script: [get-generic-python-lib](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/get-generic-python-lib)
* get,generic-python-lib,_unidecode
@@ -416,6 +435,7 @@ ___
Click here to expand this section.
* `--count=value` → `CM_MLPERF_LOADGEN_QUERY_COUNT=value`
+* `--devices=value` → `CM_MLPERF_NVIDIA_HARNESS_DEVICES=value`
* `--dla_batch_size=value` → `CM_MLPERF_NVIDIA_HARNESS_DLA_BATCH_SIZE=value`
* `--dla_copy_streams=value` → `CM_MLPERF_NVIDIA_HARNESS_DLA_COPY_STREAMS=value`
* `--dla_inference_streams=value` → `CM_MLPERF_NVIDIA_HARNESS_DLA_INFERENCE_STREAMS=value`
diff --git a/cm-mlops/script/run-mlperf-power-server/README.md b/cm-mlops/script/run-mlperf-power-server/README.md
index 12b0bdc6af..39c20644c5 100644
--- a/cm-mlops/script/run-mlperf-power-server/README.md
+++ b/cm-mlops/script/run-mlperf-power-server/README.md
@@ -111,6 +111,7 @@ ___
* `--device_type=value` → `CM_MLPERF_POWER_DEVICE_TYPE=value`
* `--interface_flag=value` → `CM_MLPERF_POWER_INTERFACE_FLAG=value`
* `--ntp_server=value` → `CM_MLPERF_POWER_NTP_SERVER=value`
+* `--screen=value` → `CM_MLPERF_POWER_SERVER_USE_SCREEN=value`
**Above CLI flags can be used in the Python CM API as follows:**
@@ -133,6 +134,7 @@ These keys can be updated via `--env.KEY=VALUE` or `env` dictionary in `@input.j
* CM_MLPERF_POWER_SERVER_ADDRESS: `0.0.0.0`
* CM_MLPERF_POWER_SERVER_PORT: `4950`
* CM_MLPERF_POWER_DEVICE_PORT: `/dev/usbtmc0`
+* CM_MLPERF_POWER_SERVER_USE_SCREEN: `no`
@@ -152,6 +154,9 @@ ___
* get,mlperf,power,daemon
* CM names: `--adr.['power-damenon']...`
- CM script: [get-spec-ptd](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/get-spec-ptd)
+ * get,generic,sys-util,_screen
+ * CM names: `--adr.['screen']...`
+ - CM script: [get-generic-sys-util](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/get-generic-sys-util)
1. ***Run "preprocess" function from [customize.py](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/run-mlperf-power-server/customize.py)***
1. Read "prehook_deps" on other CM scripts from [meta](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/run-mlperf-power-server/_cm.json)
1. ***Run native script if exists***
diff --git a/cm/README.md b/cm/README.md
index 011ba9f80a..54d8948d70 100644
--- a/cm/README.md
+++ b/cm/README.md
@@ -28,7 +28,7 @@ while reducing all the tedious, manual, repetitive, and ad-hoc efforts to reprod
CM powers the [Collective Knowledge platform (MLCommons CK playground)](https://access.cKnowledge.org)
to aggregate [reproducible experiments](https://access.cknowledge.org/playground/?action=experiments),
-connect academia and industry to [organize reproducibility and optimization challenges]( https://github.com/mlcommons/ck/tree/master/cm-mlops/challenge ),
+connect academia and industry to [organize reproducibility, replicability and optimization challenges]( https://github.com/mlcommons/ck/tree/master/cm-mlops/challenge ),
and help developers and users select Pareto-optimal end-to-end applications and systems based on their requirements and constraints
(cost, performance, power consumption, accuracy, etc).
diff --git a/cm/cmind/repo/automation/automation/README.md b/cm/cmind/repo/automation/automation/README.md
index 72e5e0600b..d2f69e84cd 100644
--- a/cm/cmind/repo/automation/automation/README.md
+++ b/cm/cmind/repo/automation/automation/README.md
@@ -2,10 +2,30 @@
### Automation actions
+#### print_input
+
+ * CM CLI: ```cm print_input automation``` ([add flags (dict keys) from this API](https://github.com/mlcommons/ck/tree/master/cm/cmind/repo/automation/automation/module.py#L15))
+ * CM CLI with UID: ```cm print_input automation,bbeb15d8f0a944a4``` ([add flags (dict keys) from this API](https://github.com/mlcommons/ck/tree/master/cm/cmind/repo/automation/automation/module.py#L15))
+ * CM Python API:
+ ```python
+ import cmind
+
+ r=cm.access({
+ 'action':'print_input'
+ 'automation':'automation,bbeb15d8f0a944a4'
+ 'out':'con'
+ ```
+ [add keys from this API](https://github.com/mlcommons/ck/tree/master/cm/cmind/repo/automation/automation/module.py#L15)
+ ```python
+ })
+ if r['return']>0:
+ print(r['error'])
+ ```
+
#### add
- * CM CLI: ```cm add automation``` ([add flags (dict keys) from this API](https://github.com/mlcommons/ck/tree/master/cm/cmind/repo/automation/automation/module.py#L15))
- * CM CLI with UID: ```cm add automation,bbeb15d8f0a944a4``` ([add flags (dict keys) from this API](https://github.com/mlcommons/ck/tree/master/cm/cmind/repo/automation/automation/module.py#L15))
+ * CM CLI: ```cm add automation``` ([add flags (dict keys) from this API](https://github.com/mlcommons/ck/tree/master/cm/cmind/repo/automation/automation/module.py#L39))
+ * CM CLI with UID: ```cm add automation,bbeb15d8f0a944a4``` ([add flags (dict keys) from this API](https://github.com/mlcommons/ck/tree/master/cm/cmind/repo/automation/automation/module.py#L39))
* CM Python API:
```python
import cmind
@@ -15,7 +35,7 @@
'automation':'automation,bbeb15d8f0a944a4'
'out':'con'
```
- [add keys from this API](https://github.com/mlcommons/ck/tree/master/cm/cmind/repo/automation/automation/module.py#L15)
+ [add keys from this API](https://github.com/mlcommons/ck/tree/master/cm/cmind/repo/automation/automation/module.py#L39)
```python
})
if r['return']>0:
@@ -24,8 +44,8 @@
#### doc
- * CM CLI: ```cm doc automation``` ([add flags (dict keys) from this API](https://github.com/mlcommons/ck/tree/master/cm/cmind/repo/automation/automation/module.py#L87))
- * CM CLI with UID: ```cm doc automation,bbeb15d8f0a944a4``` ([add flags (dict keys) from this API](https://github.com/mlcommons/ck/tree/master/cm/cmind/repo/automation/automation/module.py#L87))
+ * CM CLI: ```cm doc automation``` ([add flags (dict keys) from this API](https://github.com/mlcommons/ck/tree/master/cm/cmind/repo/automation/automation/module.py#L111))
+ * CM CLI with UID: ```cm doc automation,bbeb15d8f0a944a4``` ([add flags (dict keys) from this API](https://github.com/mlcommons/ck/tree/master/cm/cmind/repo/automation/automation/module.py#L111))
* CM Python API:
```python
import cmind
@@ -35,7 +55,7 @@
'automation':'automation,bbeb15d8f0a944a4'
'out':'con'
```
- [add keys from this API](https://github.com/mlcommons/ck/tree/master/cm/cmind/repo/automation/automation/module.py#L87)
+ [add keys from this API](https://github.com/mlcommons/ck/tree/master/cm/cmind/repo/automation/automation/module.py#L111)
```python
})
if r['return']>0:
@@ -44,4 +64,4 @@
### Maintainers
-* [Open MLCommons taskforce on automation and reproducibility](https://github.com/mlcommons/ck/blob/master/docs/mlperf-education-workgroup.md)
\ No newline at end of file
+* [Open MLCommons taskforce on automation and reproducibility](https://cKnowledge.org/mlcommons-taskforce)
\ No newline at end of file
diff --git a/cm/cmind/repo/automation/ck/README.md b/cm/cmind/repo/automation/ck/README.md
index 426354e3d2..97d6969835 100644
--- a/cm/cmind/repo/automation/ck/README.md
+++ b/cm/cmind/repo/automation/ck/README.md
@@ -24,4 +24,4 @@
### Maintainers
-* [Open MLCommons taskforce on automation and reproducibility](https://github.com/mlcommons/ck/blob/master/docs/mlperf-education-workgroup.md)
\ No newline at end of file
+* [Open MLCommons taskforce on automation and reproducibility](https://cKnowledge.org/mlcommons-taskforce)
\ No newline at end of file
diff --git a/cm/cmind/repo/automation/core/README.md b/cm/cmind/repo/automation/core/README.md
index f77121f8a8..7a4c099fd3 100644
--- a/cm/cmind/repo/automation/core/README.md
+++ b/cm/cmind/repo/automation/core/README.md
@@ -24,4 +24,4 @@
### Maintainers
-* [Open MLCommons taskforce on automation and reproducibility](https://github.com/mlcommons/ck/blob/master/docs/mlperf-education-workgroup.md)
\ No newline at end of file
+* [Open MLCommons taskforce on automation and reproducibility](https://cKnowledge.org/mlcommons-taskforce)
\ No newline at end of file
diff --git a/cm/cmind/repo/automation/repo/README.md b/cm/cmind/repo/automation/repo/README.md
index 7fec7a8420..4abc8c048e 100644
--- a/cm/cmind/repo/automation/repo/README.md
+++ b/cm/cmind/repo/automation/repo/README.md
@@ -24,8 +24,8 @@
#### search
- * CM CLI: ```cm search repo``` ([add flags (dict keys) from this API](https://github.com/mlcommons/ck/tree/master/cm/cmind/repo/automation/repo/module.py#L93))
- * CM CLI with UID: ```cm search repo,55c3e27e8a140e48``` ([add flags (dict keys) from this API](https://github.com/mlcommons/ck/tree/master/cm/cmind/repo/automation/repo/module.py#L93))
+ * CM CLI: ```cm search repo``` ([add flags (dict keys) from this API](https://github.com/mlcommons/ck/tree/master/cm/cmind/repo/automation/repo/module.py#L98))
+ * CM CLI with UID: ```cm search repo,55c3e27e8a140e48``` ([add flags (dict keys) from this API](https://github.com/mlcommons/ck/tree/master/cm/cmind/repo/automation/repo/module.py#L98))
* CM Python API:
```python
import cmind
@@ -35,7 +35,7 @@
'automation':'repo,55c3e27e8a140e48'
'out':'con'
```
- [add keys from this API](https://github.com/mlcommons/ck/tree/master/cm/cmind/repo/automation/repo/module.py#L93)
+ [add keys from this API](https://github.com/mlcommons/ck/tree/master/cm/cmind/repo/automation/repo/module.py#L98)
```python
})
if r['return']>0:
@@ -44,8 +44,8 @@
#### update
- * CM CLI: ```cm update repo``` ([add flags (dict keys) from this API](https://github.com/mlcommons/ck/tree/master/cm/cmind/repo/automation/repo/module.py#L172))
- * CM CLI with UID: ```cm update repo,55c3e27e8a140e48``` ([add flags (dict keys) from this API](https://github.com/mlcommons/ck/tree/master/cm/cmind/repo/automation/repo/module.py#L172))
+ * CM CLI: ```cm update repo``` ([add flags (dict keys) from this API](https://github.com/mlcommons/ck/tree/master/cm/cmind/repo/automation/repo/module.py#L177))
+ * CM CLI with UID: ```cm update repo,55c3e27e8a140e48``` ([add flags (dict keys) from this API](https://github.com/mlcommons/ck/tree/master/cm/cmind/repo/automation/repo/module.py#L177))
* CM Python API:
```python
import cmind
@@ -55,7 +55,7 @@
'automation':'repo,55c3e27e8a140e48'
'out':'con'
```
- [add keys from this API](https://github.com/mlcommons/ck/tree/master/cm/cmind/repo/automation/repo/module.py#L172)
+ [add keys from this API](https://github.com/mlcommons/ck/tree/master/cm/cmind/repo/automation/repo/module.py#L177)
```python
})
if r['return']>0:
@@ -64,8 +64,8 @@
#### delete
- * CM CLI: ```cm delete repo``` ([add flags (dict keys) from this API](https://github.com/mlcommons/ck/tree/master/cm/cmind/repo/automation/repo/module.py#L209))
- * CM CLI with UID: ```cm delete repo,55c3e27e8a140e48``` ([add flags (dict keys) from this API](https://github.com/mlcommons/ck/tree/master/cm/cmind/repo/automation/repo/module.py#L209))
+ * CM CLI: ```cm delete repo``` ([add flags (dict keys) from this API](https://github.com/mlcommons/ck/tree/master/cm/cmind/repo/automation/repo/module.py#L214))
+ * CM CLI with UID: ```cm delete repo,55c3e27e8a140e48``` ([add flags (dict keys) from this API](https://github.com/mlcommons/ck/tree/master/cm/cmind/repo/automation/repo/module.py#L214))
* CM Python API:
```python
import cmind
@@ -75,7 +75,7 @@
'automation':'repo,55c3e27e8a140e48'
'out':'con'
```
- [add keys from this API](https://github.com/mlcommons/ck/tree/master/cm/cmind/repo/automation/repo/module.py#L209)
+ [add keys from this API](https://github.com/mlcommons/ck/tree/master/cm/cmind/repo/automation/repo/module.py#L214)
```python
})
if r['return']>0:
@@ -84,8 +84,8 @@
#### init
- * CM CLI: ```cm init repo``` ([add flags (dict keys) from this API](https://github.com/mlcommons/ck/tree/master/cm/cmind/repo/automation/repo/module.py#L262))
- * CM CLI with UID: ```cm init repo,55c3e27e8a140e48``` ([add flags (dict keys) from this API](https://github.com/mlcommons/ck/tree/master/cm/cmind/repo/automation/repo/module.py#L262))
+ * CM CLI: ```cm init repo``` ([add flags (dict keys) from this API](https://github.com/mlcommons/ck/tree/master/cm/cmind/repo/automation/repo/module.py#L271))
+ * CM CLI with UID: ```cm init repo,55c3e27e8a140e48``` ([add flags (dict keys) from this API](https://github.com/mlcommons/ck/tree/master/cm/cmind/repo/automation/repo/module.py#L271))
* CM Python API:
```python
import cmind
@@ -95,7 +95,7 @@
'automation':'repo,55c3e27e8a140e48'
'out':'con'
```
- [add keys from this API](https://github.com/mlcommons/ck/tree/master/cm/cmind/repo/automation/repo/module.py#L262)
+ [add keys from this API](https://github.com/mlcommons/ck/tree/master/cm/cmind/repo/automation/repo/module.py#L271)
```python
})
if r['return']>0:
@@ -104,8 +104,8 @@
#### add
- * CM CLI: ```cm add repo``` ([add flags (dict keys) from this API](https://github.com/mlcommons/ck/tree/master/cm/cmind/repo/automation/repo/module.py#L381))
- * CM CLI with UID: ```cm add repo,55c3e27e8a140e48``` ([add flags (dict keys) from this API](https://github.com/mlcommons/ck/tree/master/cm/cmind/repo/automation/repo/module.py#L381))
+ * CM CLI: ```cm add repo``` ([add flags (dict keys) from this API](https://github.com/mlcommons/ck/tree/master/cm/cmind/repo/automation/repo/module.py#L395))
+ * CM CLI with UID: ```cm add repo,55c3e27e8a140e48``` ([add flags (dict keys) from this API](https://github.com/mlcommons/ck/tree/master/cm/cmind/repo/automation/repo/module.py#L395))
* CM Python API:
```python
import cmind
@@ -115,7 +115,7 @@
'automation':'repo,55c3e27e8a140e48'
'out':'con'
```
- [add keys from this API](https://github.com/mlcommons/ck/tree/master/cm/cmind/repo/automation/repo/module.py#L381)
+ [add keys from this API](https://github.com/mlcommons/ck/tree/master/cm/cmind/repo/automation/repo/module.py#L395)
```python
})
if r['return']>0:
@@ -124,8 +124,8 @@
#### pack
- * CM CLI: ```cm pack repo``` ([add flags (dict keys) from this API](https://github.com/mlcommons/ck/tree/master/cm/cmind/repo/automation/repo/module.py#L389))
- * CM CLI with UID: ```cm pack repo,55c3e27e8a140e48``` ([add flags (dict keys) from this API](https://github.com/mlcommons/ck/tree/master/cm/cmind/repo/automation/repo/module.py#L389))
+ * CM CLI: ```cm pack repo``` ([add flags (dict keys) from this API](https://github.com/mlcommons/ck/tree/master/cm/cmind/repo/automation/repo/module.py#L403))
+ * CM CLI with UID: ```cm pack repo,55c3e27e8a140e48``` ([add flags (dict keys) from this API](https://github.com/mlcommons/ck/tree/master/cm/cmind/repo/automation/repo/module.py#L403))
* CM Python API:
```python
import cmind
@@ -135,7 +135,7 @@
'automation':'repo,55c3e27e8a140e48'
'out':'con'
```
- [add keys from this API](https://github.com/mlcommons/ck/tree/master/cm/cmind/repo/automation/repo/module.py#L389)
+ [add keys from this API](https://github.com/mlcommons/ck/tree/master/cm/cmind/repo/automation/repo/module.py#L403)
```python
})
if r['return']>0:
@@ -144,8 +144,8 @@
#### unpack
- * CM CLI: ```cm unpack repo``` ([add flags (dict keys) from this API](https://github.com/mlcommons/ck/tree/master/cm/cmind/repo/automation/repo/module.py#L459))
- * CM CLI with UID: ```cm unpack repo,55c3e27e8a140e48``` ([add flags (dict keys) from this API](https://github.com/mlcommons/ck/tree/master/cm/cmind/repo/automation/repo/module.py#L459))
+ * CM CLI: ```cm unpack repo``` ([add flags (dict keys) from this API](https://github.com/mlcommons/ck/tree/master/cm/cmind/repo/automation/repo/module.py#L473))
+ * CM CLI with UID: ```cm unpack repo,55c3e27e8a140e48``` ([add flags (dict keys) from this API](https://github.com/mlcommons/ck/tree/master/cm/cmind/repo/automation/repo/module.py#L473))
* CM Python API:
```python
import cmind
@@ -155,7 +155,7 @@
'automation':'repo,55c3e27e8a140e48'
'out':'con'
```
- [add keys from this API](https://github.com/mlcommons/ck/tree/master/cm/cmind/repo/automation/repo/module.py#L459)
+ [add keys from this API](https://github.com/mlcommons/ck/tree/master/cm/cmind/repo/automation/repo/module.py#L473)
```python
})
if r['return']>0:
@@ -164,8 +164,8 @@
#### import_ck_to_cm
- * CM CLI: ```cm import_ck_to_cm repo``` ([add flags (dict keys) from this API](https://github.com/mlcommons/ck/tree/master/cm/cmind/repo/automation/repo/module.py#L562))
- * CM CLI with UID: ```cm import_ck_to_cm repo,55c3e27e8a140e48``` ([add flags (dict keys) from this API](https://github.com/mlcommons/ck/tree/master/cm/cmind/repo/automation/repo/module.py#L562))
+ * CM CLI: ```cm import_ck_to_cm repo``` ([add flags (dict keys) from this API](https://github.com/mlcommons/ck/tree/master/cm/cmind/repo/automation/repo/module.py#L580))
+ * CM CLI with UID: ```cm import_ck_to_cm repo,55c3e27e8a140e48``` ([add flags (dict keys) from this API](https://github.com/mlcommons/ck/tree/master/cm/cmind/repo/automation/repo/module.py#L580))
* CM Python API:
```python
import cmind
@@ -175,7 +175,7 @@
'automation':'repo,55c3e27e8a140e48'
'out':'con'
```
- [add keys from this API](https://github.com/mlcommons/ck/tree/master/cm/cmind/repo/automation/repo/module.py#L562)
+ [add keys from this API](https://github.com/mlcommons/ck/tree/master/cm/cmind/repo/automation/repo/module.py#L580)
```python
})
if r['return']>0:
@@ -184,8 +184,8 @@
#### convert_ck_to_cm
- * CM CLI: ```cm convert_ck_to_cm repo``` ([add flags (dict keys) from this API](https://github.com/mlcommons/ck/tree/master/cm/cmind/repo/automation/repo/module.py#L613))
- * CM CLI with UID: ```cm convert_ck_to_cm repo,55c3e27e8a140e48``` ([add flags (dict keys) from this API](https://github.com/mlcommons/ck/tree/master/cm/cmind/repo/automation/repo/module.py#L613))
+ * CM CLI: ```cm convert_ck_to_cm repo``` ([add flags (dict keys) from this API](https://github.com/mlcommons/ck/tree/master/cm/cmind/repo/automation/repo/module.py#L631))
+ * CM CLI with UID: ```cm convert_ck_to_cm repo,55c3e27e8a140e48``` ([add flags (dict keys) from this API](https://github.com/mlcommons/ck/tree/master/cm/cmind/repo/automation/repo/module.py#L631))
* CM Python API:
```python
import cmind
@@ -195,7 +195,7 @@
'automation':'repo,55c3e27e8a140e48'
'out':'con'
```
- [add keys from this API](https://github.com/mlcommons/ck/tree/master/cm/cmind/repo/automation/repo/module.py#L613)
+ [add keys from this API](https://github.com/mlcommons/ck/tree/master/cm/cmind/repo/automation/repo/module.py#L631)
```python
})
if r['return']>0:
@@ -204,8 +204,8 @@
#### detect
- * CM CLI: ```cm detect repo``` ([add flags (dict keys) from this API](https://github.com/mlcommons/ck/tree/master/cm/cmind/repo/automation/repo/module.py#L667))
- * CM CLI with UID: ```cm detect repo,55c3e27e8a140e48``` ([add flags (dict keys) from this API](https://github.com/mlcommons/ck/tree/master/cm/cmind/repo/automation/repo/module.py#L667))
+ * CM CLI: ```cm detect repo``` ([add flags (dict keys) from this API](https://github.com/mlcommons/ck/tree/master/cm/cmind/repo/automation/repo/module.py#L685))
+ * CM CLI with UID: ```cm detect repo,55c3e27e8a140e48``` ([add flags (dict keys) from this API](https://github.com/mlcommons/ck/tree/master/cm/cmind/repo/automation/repo/module.py#L685))
* CM Python API:
```python
import cmind
@@ -215,7 +215,27 @@
'automation':'repo,55c3e27e8a140e48'
'out':'con'
```
- [add keys from this API](https://github.com/mlcommons/ck/tree/master/cm/cmind/repo/automation/repo/module.py#L667)
+ [add keys from this API](https://github.com/mlcommons/ck/tree/master/cm/cmind/repo/automation/repo/module.py#L685)
+ ```python
+ })
+ if r['return']>0:
+ print(r['error'])
+ ```
+
+#### reindex
+
+ * CM CLI: ```cm reindex repo``` ([add flags (dict keys) from this API](https://github.com/mlcommons/ck/tree/master/cm/cmind/repo/automation/repo/module.py#L872))
+ * CM CLI with UID: ```cm reindex repo,55c3e27e8a140e48``` ([add flags (dict keys) from this API](https://github.com/mlcommons/ck/tree/master/cm/cmind/repo/automation/repo/module.py#L872))
+ * CM Python API:
+ ```python
+ import cmind
+
+ r=cm.access({
+ 'action':'reindex'
+ 'automation':'repo,55c3e27e8a140e48'
+ 'out':'con'
+ ```
+ [add keys from this API](https://github.com/mlcommons/ck/tree/master/cm/cmind/repo/automation/repo/module.py#L872)
```python
})
if r['return']>0:
@@ -224,4 +244,4 @@
### Maintainers
-* [Open MLCommons taskforce on automation and reproducibility](https://github.com/mlcommons/ck/blob/master/docs/mlperf-education-workgroup.md)
\ No newline at end of file
+* [Open MLCommons taskforce on automation and reproducibility](https://cKnowledge.org/mlcommons-taskforce)
\ No newline at end of file
diff --git a/docs/_generator/generate_toc.cmd b/docs/_generator/generate_toc.cmd
index 626eee263b..4d3d9b4725 100644
--- a/docs/_generator/generate_toc.cmd
+++ b/docs/_generator/generate_toc.cmd
@@ -6,10 +6,15 @@ cm create-toc-from-md utils --input=sc22-scc-mlperf-part3.md
cm create-toc-from-md utils --input=mlperf-inference-submission.md
cm create-toc-from-md utils --input=concept.md
+cd ../artifact-evaluation
+
+cm create-toc-from-md utils --input=faq.md
+
cd ../
cm create-toc-from-md utils --input=taskforce.md
cm create-toc-from-md utils --input=installation.md
+cm create-toc-from-md utils --input=faq.md
cm create-toc-from-md utils --input=README.md
cd ../
diff --git a/docs/artifact-evaluation/faq.md b/docs/artifact-evaluation/faq.md
index 4f853302e3..7712b1f7a4 100644
--- a/docs/artifact-evaluation/faq.md
+++ b/docs/artifact-evaluation/faq.md
@@ -1,4 +1,27 @@
-# Artifact evaluation
+# Artifact Evaluation FAQ
+
+
+Click here to see the table of contents.
+
+* [Artifact evaluation](#artifact-evaluation)
+ * [Frequently Asked Questions](#frequently-asked-questions)
+ * [What is the difference between Repeatability, Reproducibility and Replicability?](#what-is-the-difference-between-repeatability-reproducibility-and-replicability?)
+ * [Do I have to open source my software artifacts?](#do-i-have-to-open-source-my-software-artifacts?)
+ * [Is Artifact evaluation blind or double-blind?](#is-artifact-evaluation-blind-or-double-blind?)
+ * [How to pack artifacts?](#how-to-pack-artifacts?)
+ * [Is it possible to provide a remote access to a machine with pre-installed artifacts?](#is-it-possible-to-provide-a-remote-access-to-a-machine-with-pre-installed-artifacts?)
+ * [Can I share commercial benchmarks or software with evaluators?](#can-i-share-commercial-benchmarks-or-software-with-evaluators?)
+ * [Can I engage with the community to evaluate my artifacts?](#can-i-engage-with-the-community-to-evaluate-my-artifacts?)
+ * [How to automate, customize and port experiments?](#how-to-automate-customize-and-port-experiments?)
+ * [Do I have to make my artifacts public if they pass evaluation?](#do-i-have-to-make-my-artifacts-public-if-they-pass-evaluation?)
+ * [How to report and compare empirical results?](#how-to-report-and-compare-empirical-results?)
+ * [How to deal with numerical accuracy and instability?](#how-to-deal-with-numerical-accuracy-and-instability?)
+ * [How to validate models or algorithm scalability?](#how-to-validate-models-or-algorithm-scalability?)
+ * [Is there any page limit for my Artifact Evaluation Appendix?](#is-there-any-page-limit-for-my-artifact-evaluation-appendix?)
+ * [Where can I find a sample HotCRP configuration to set up AE?](#where-can-i-find-a-sample-hotcrp-configuration-to-set-up-ae?)
+ * [Questions and Feedback](#questions-and-feedback)
+
+
## Frequently Asked Questions
@@ -7,6 +30,31 @@
to post them to the dedicated [AE google group](https://groups.google.com/forum/#!forum/artifact-evaluation)
or contact the [open MLCommons taskforce on automation and reproducibility](https://github.com/mlcommons/ck/blob/master/docs/taskforce.md).**
+
+### What is the difference between Repeatability, Reproducibility and Replicability?
+
+We use the following definitions [adopted by ACM and NISO](https://www.acm.org/publications/policies/artifact-review-badging):
+
+* *Repeatability (Same team, same experimental setup)*
+
+ The measurement can be obtained with stated precision by the same team using the same measurement procedure,
+ the same measuring system, under the same operating conditions, in the same location on multiple trials.
+ For computational experiments, this means that a researcher can reliably repeat her own computation.
+
+* *Reproducibility (Different team, different experimental setup)*
+
+ The measurement can be obtained with stated precision by a different team using the same measurement procedure,
+ the same measuring system, under the same operating conditions, in the same or a different location on multiple trials.
+ For computational experiments, this means that an independent group can obtain the same result using the author's own artifacts.
+
+* *Replicability (Different team, same experimental setup)*
+
+ The measurement can be obtained with stated precision by a different team, a different measuring system,
+ in a different location on multiple trials. For computational experiments, this means that an independent group
+ can obtain the same result using artifacts which they develop completely independently.
+
+
+
### Do I have to open source my software artifacts?
diff --git a/docs/faq.md b/docs/faq.md
index d6733ed10d..3692a67394 100644
--- a/docs/faq.md
+++ b/docs/faq.md
@@ -1,8 +1,50 @@
[ [Back to index](README.md) ]
-## CM FAQ
+## CM/CK FAQ
-To be added...
+
+Click here to see the table of contents.
-Feel free to ask your questions at our [public Discord server](https://discord.gg/JjWNWXKxwT).
+ * [CM FAQ](#cm-faq)
+ * [How to use CM scripts without affecting native Python installation?](#how-to-use-cm-scripts-without-affecting-native-python-installation?)
+ * [What is the difference between Repeatability, Reproducibility and Replicability?](#what-is-the-difference-between-repeatability-reproducibility-and-replicability?)
+ * [Discussions](#discussions)
+
+
+
+
+### How to use CM scripts without affecting native Python installation?
+
+We've created CM automation to help users set up multiple virtual Python environments
+without altering current Python installation. Please follow [this guide](../cm-mlops/automation/script/README-extra.md#using-python-virtual-environments)
+to set up Python virtual environment on your system.
+
+### What is the difference between Repeatability, Reproducibility and Replicability?
+
+We use the following definitions [adopted by ACM and NISO](https://www.acm.org/publications/policies/artifact-review-badging):
+
+* *Repeatability (Same team, same experimental setup)*
+
+ The measurement can be obtained with stated precision by the same team using the same measurement procedure,
+ the same measuring system, under the same operating conditions, in the same location on multiple trials.
+ For computational experiments, this means that a researcher can reliably repeat her own computation.
+* *Reproducibility (Different team, different experimental setup)*
+
+ The measurement can be obtained with stated precision by a different team using the same measurement procedure,
+ the same measuring system, under the same operating conditions, in the same or a different location on multiple trials.
+ For computational experiments, this means that an independent group can obtain the same result using the author's own artifacts.
+
+* *Replicability (Different team, same experimental setup)*
+
+ The measurement can be obtained with stated precision by a different team, a different measuring system,
+ in a different location on multiple trials. For computational experiments, this means that an independent group
+ can obtain the same result using artifacts which they develop completely independently.
+
+
+
+## Discussions
+
+Feel free to ask your questions at our [public Discord server](https://discord.gg/JjWNWXKxwT).
+Join our [MLCommons taskforce on automation and reproducibility](taskforce.md) to participate
+in collaborative developments and optimization challenges.
diff --git a/docs/installation.md b/docs/installation.md
index c00a6b08e7..98a57416f1 100644
--- a/docs/installation.md
+++ b/docs/installation.md
@@ -156,11 +156,19 @@ You can use the following environment variables to customize CM installation:
# CM automation scripts
-You can use [CM automation scripts](https://github.com/mlcommons/ck/tree/master/cm-mlops/script)
-for portable MLOps and DevOps from MLCommons by installing the following repository:
+Please go [back to index](README.md) to continue learning about CM interface and scripts.
+
+However, if you are already familiar with the CM/CK concepts, you can
+use these [CM automation scripts](https://github.com/mlcommons/ck/tree/master/cm-mlops/script)
+for portable MLOps and DevOps from MLCommons directly by installing the following repository:
```bash
cm pull repo mlcommons@ck
```
If you use CM scripts with Python outside containers, we suggest you to set up CM Python virtual
environment as described [here](../cm-mlops/automation/script/README-extra.md#using-python-virtual-environments).
+
+Feel free to check [these CM tutorials](tutorials) to learn how to use CM to facilitate reproducible research,
+run MLPerf out-of-the-box and accelerate technology transfer across rapidly evolving
+software, hardware, models and data.
+
diff --git a/docs/introduction-ck.md b/docs/introduction-ck.md
index 5c34a6a803..99f3a17eea 100644
--- a/docs/introduction-ck.md
+++ b/docs/introduction-ck.md
@@ -16,7 +16,7 @@ based on their requirements and constraints (optimal trade off between performan
See a few real-world examples of using the CK playground powered by the CM language:
-- [Organizing reproducibility and optimization challenges](https://access.cknowledge.org/playground/?action=challenges&name=57cbc3384d7640f9)
+- [Organizing reproducibility, replicability and optimization challenges](https://access.cknowledge.org/playground/?action=challenges&name=57cbc3384d7640f9)
- [Reproducing MLPerf inference benchmark and automating submissions](https://cKnowledge.org/mlperf-inference-gui)
- [Visualizing and comparing MLPerf inference benchmark results](https://access.cKnowledge.org/playground/?action=experiments&tags=mlperf-inference,all,open,edge,image-classification,singlestream)
- [Sharing reproducibility reports]( https://cKnowledge.org/mlperf-inf-v3.0-reproducibility-report )
diff --git a/docs/list_of_automations.md b/docs/list_of_automations.md
index b092a83a2f..0836b03232 100644
--- a/docs/list_of_automations.md
+++ b/docs/list_of_automations.md
@@ -6,14 +6,17 @@
This file is generated automatically - don't edit!
-->
-* [repo](#repo) *(Managing CM repositories)*
+* [repo](#repo) *(Managing CM repositories and software projects)*
* [script](#script) *(Making native scripts more portable, interoperable and deterministic)*
* [cache](#cache) *(Caching cross-platform CM scripts)*
* [utils](#utils) *(Accessing various CM utils)*
* [core](#core) *(Accessing some core CM functions)*
+* [challenge](#challenge)
+* [contributor](#contributor)
* [docker](#docker) *(Managing modular docker containers (under development))*
* [experiment](#experiment) *(Managing and reproducing experiments (under development))*
* [project](#project)
+* [report](#report)
* [ck](#ck) *(Accessing legacy CK automations)*
* [automation](#automation) *(Managing CM automations)*
@@ -21,23 +24,24 @@ This file is generated automatically - don't edit!
## repo
-*Managing CM repositories.*
+*Managing CM repositories and software projects.*
* GitHub repository with CM automations: *cm pull [internal](https://github.com/mlcommons/ck/tree/master/cm/cmind/repo)*
* CM automation code and meta: *[GitHub](https://github.com/mlcommons/ck/tree/master/cm/cmind/repo/automation/repo)*
* CM automation actions:
* cm **pull** repo *( [See CM API](https://github.com/mlcommons/ck/tree/master/cm/cmind/repo/automation/repo/module.py#L15) )*
- * cm **search** repo *( [See CM API](https://github.com/mlcommons/ck/tree/master/cm/cmind/repo/automation/repo/module.py#L93) )*
- * cm **update** repo *( [See CM API](https://github.com/mlcommons/ck/tree/master/cm/cmind/repo/automation/repo/module.py#L172) )*
- * cm **delete** repo *( [See CM API](https://github.com/mlcommons/ck/tree/master/cm/cmind/repo/automation/repo/module.py#L209) )*
- * cm **init** repo *( [See CM API](https://github.com/mlcommons/ck/tree/master/cm/cmind/repo/automation/repo/module.py#L262) )*
- * cm **add** repo *( [See CM API](https://github.com/mlcommons/ck/tree/master/cm/cmind/repo/automation/repo/module.py#L381) )*
- * cm **pack** repo *( [See CM API](https://github.com/mlcommons/ck/tree/master/cm/cmind/repo/automation/repo/module.py#L389) )*
- * cm **unpack** repo *( [See CM API](https://github.com/mlcommons/ck/tree/master/cm/cmind/repo/automation/repo/module.py#L459) )*
- * cm **import_ck_to_cm** repo *( [See CM API](https://github.com/mlcommons/ck/tree/master/cm/cmind/repo/automation/repo/module.py#L562) )*
- * cm **convert_ck_to_cm** repo *( [See CM API](https://github.com/mlcommons/ck/tree/master/cm/cmind/repo/automation/repo/module.py#L613) )*
- * cm **detect** repo *( [See CM API](https://github.com/mlcommons/ck/tree/master/cm/cmind/repo/automation/repo/module.py#L667) )*
+ * cm **search** repo *( [See CM API](https://github.com/mlcommons/ck/tree/master/cm/cmind/repo/automation/repo/module.py#L98) )*
+ * cm **update** repo *( [See CM API](https://github.com/mlcommons/ck/tree/master/cm/cmind/repo/automation/repo/module.py#L177) )*
+ * cm **delete** repo *( [See CM API](https://github.com/mlcommons/ck/tree/master/cm/cmind/repo/automation/repo/module.py#L214) )*
+ * cm **init** repo *( [See CM API](https://github.com/mlcommons/ck/tree/master/cm/cmind/repo/automation/repo/module.py#L271) )*
+ * cm **add** repo *( [See CM API](https://github.com/mlcommons/ck/tree/master/cm/cmind/repo/automation/repo/module.py#L395) )*
+ * cm **pack** repo *( [See CM API](https://github.com/mlcommons/ck/tree/master/cm/cmind/repo/automation/repo/module.py#L403) )*
+ * cm **unpack** repo *( [See CM API](https://github.com/mlcommons/ck/tree/master/cm/cmind/repo/automation/repo/module.py#L473) )*
+ * cm **import_ck_to_cm** repo *( [See CM API](https://github.com/mlcommons/ck/tree/master/cm/cmind/repo/automation/repo/module.py#L580) )*
+ * cm **convert_ck_to_cm** repo *( [See CM API](https://github.com/mlcommons/ck/tree/master/cm/cmind/repo/automation/repo/module.py#L631) )*
+ * cm **detect** repo *( [See CM API](https://github.com/mlcommons/ck/tree/master/cm/cmind/repo/automation/repo/module.py#L685) )*
+ * cm **reindex** repo *( [See CM API](https://github.com/mlcommons/ck/tree/master/cm/cmind/repo/automation/repo/module.py#L872) )*
## script
@@ -50,21 +54,23 @@ This file is generated automatically - don't edit!
* CM automation code and meta: *[GitHub](https://github.com/mlcommons/ck/tree/master/cm-mlops/automation/script)*
* CM automation actions:
* cm **run** script *( [See CM API](https://github.com/mlcommons/ck/tree/master/cm-mlops/automation/script/module.py#L72) )*
- * cm **version** script *( [See CM API](https://github.com/mlcommons/ck/tree/master/cm-mlops/automation/script/module.py#L1656) )*
- * cm **search** script *( [See CM API](https://github.com/mlcommons/ck/tree/master/cm-mlops/automation/script/module.py#L1684) )*
- * cm **test** script *( [See CM API](https://github.com/mlcommons/ck/tree/master/cm-mlops/automation/script/module.py#L1792) )*
- * cm **native_run** script *( [See CM API](https://github.com/mlcommons/ck/tree/master/cm-mlops/automation/script/module.py#L1858) )*
- * cm **add** script *( [See CM API](https://github.com/mlcommons/ck/tree/master/cm-mlops/automation/script/module.py#L1931) )*
- * cm **run_native_script** script *( [See CM API](https://github.com/mlcommons/ck/tree/master/cm-mlops/automation/script/module.py#L2399) )*
- * cm **find_file_in_paths** script *( [See CM API](https://github.com/mlcommons/ck/tree/master/cm-mlops/automation/script/module.py#L2440) )*
- * cm **detect_version_using_script** script *( [See CM API](https://github.com/mlcommons/ck/tree/master/cm-mlops/automation/script/module.py#L2654) )*
- * cm **find_artifact** script *( [See CM API](https://github.com/mlcommons/ck/tree/master/cm-mlops/automation/script/module.py#L2727) )*
- * cm **find_file_deep** script *( [See CM API](https://github.com/mlcommons/ck/tree/master/cm-mlops/automation/script/module.py#L2885) )*
- * cm **find_file_back** script *( [See CM API](https://github.com/mlcommons/ck/tree/master/cm-mlops/automation/script/module.py#L2943) )*
- * cm **parse_version** script *( [See CM API](https://github.com/mlcommons/ck/tree/master/cm-mlops/automation/script/module.py#L2984) )*
- * cm **update_deps** script *( [See CM API](https://github.com/mlcommons/ck/tree/master/cm-mlops/automation/script/module.py#L3038) )*
- * cm **get_default_path_list** script *( [See CM API](https://github.com/mlcommons/ck/tree/master/cm-mlops/automation/script/module.py#L3057) )*
- * cm **doc** script *( [See CM API](https://github.com/mlcommons/ck/tree/master/cm-mlops/automation/script/module.py#L3068) )*
+ * cm **version** script *( [See CM API](https://github.com/mlcommons/ck/tree/master/cm-mlops/automation/script/module.py#L1697) )*
+ * cm **search** script *( [See CM API](https://github.com/mlcommons/ck/tree/master/cm-mlops/automation/script/module.py#L1725) )*
+ * cm **test** script *( [See CM API](https://github.com/mlcommons/ck/tree/master/cm-mlops/automation/script/module.py#L1835) )*
+ * cm **native_run** script *( [See CM API](https://github.com/mlcommons/ck/tree/master/cm-mlops/automation/script/module.py#L1901) )*
+ * cm **add** script *( [See CM API](https://github.com/mlcommons/ck/tree/master/cm-mlops/automation/script/module.py#L1974) )*
+ * cm **run_native_script** script *( [See CM API](https://github.com/mlcommons/ck/tree/master/cm-mlops/automation/script/module.py#L2463) )*
+ * cm **find_file_in_paths** script *( [See CM API](https://github.com/mlcommons/ck/tree/master/cm-mlops/automation/script/module.py#L2504) )*
+ * cm **detect_version_using_script** script *( [See CM API](https://github.com/mlcommons/ck/tree/master/cm-mlops/automation/script/module.py#L2718) )*
+ * cm **find_artifact** script *( [See CM API](https://github.com/mlcommons/ck/tree/master/cm-mlops/automation/script/module.py#L2791) )*
+ * cm **find_file_deep** script *( [See CM API](https://github.com/mlcommons/ck/tree/master/cm-mlops/automation/script/module.py#L2949) )*
+ * cm **find_file_back** script *( [See CM API](https://github.com/mlcommons/ck/tree/master/cm-mlops/automation/script/module.py#L3007) )*
+ * cm **parse_version** script *( [See CM API](https://github.com/mlcommons/ck/tree/master/cm-mlops/automation/script/module.py#L3048) )*
+ * cm **update_deps** script *( [See CM API](https://github.com/mlcommons/ck/tree/master/cm-mlops/automation/script/module.py#L3102) )*
+ * cm **get_default_path_list** script *( [See CM API](https://github.com/mlcommons/ck/tree/master/cm-mlops/automation/script/module.py#L3121) )*
+ * cm **doc** script *( [See CM API](https://github.com/mlcommons/ck/tree/master/cm-mlops/automation/script/module.py#L3132) )*
+ * cm **dockerfile** script *( [See CM API](https://github.com/mlcommons/ck/tree/master/cm-mlops/automation/script/module.py#L3161) )*
+ * cm **docker** script *( [See CM API](https://github.com/mlcommons/ck/tree/master/cm-mlops/automation/script/module.py#L3189) )*
## cache
@@ -115,6 +121,24 @@ This file is generated automatically - don't edit!
* cm **uid** core *( [See CM API](https://github.com/mlcommons/ck/tree/master/cm/cmind/repo/automation/core/module.py#L22) )*
+## challenge
+
+
+* GitHub repository with CM automations: *cm pull [mlcommons@ck](https://github.com/mlcommons/ck/tree/master/cm-mlops)*
+* CM automation code and meta: *[GitHub](https://github.com/mlcommons/ck/tree/master/cm-mlops/automation/challenge)*
+* CM automation actions:
+ * cm **test** challenge *( [See CM API](https://github.com/mlcommons/ck/tree/master/cm-mlops/automation/challenge/module.py#L15) )*
+
+
+## contributor
+
+
+* GitHub repository with CM automations: *cm pull [mlcommons@ck](https://github.com/mlcommons/ck/tree/master/cm-mlops)*
+* CM automation code and meta: *[GitHub](https://github.com/mlcommons/ck/tree/master/cm-mlops/automation/contributor)*
+* CM automation actions:
+ * cm **test** contributor *( [See CM API](https://github.com/mlcommons/ck/tree/master/cm-mlops/automation/contributor/module.py#L15) )*
+
+
## docker
@@ -150,6 +174,15 @@ This file is generated automatically - don't edit!
* cm **test** project *( [See CM API](https://github.com/mlcommons/ck/tree/master/cm-mlops/automation/project/module.py#L15) )*
+## report
+
+
+* GitHub repository with CM automations: *cm pull [mlcommons@ck](https://github.com/mlcommons/ck/tree/master/cm-mlops)*
+* CM automation code and meta: *[GitHub](https://github.com/mlcommons/ck/tree/master/cm-mlops/automation/report)*
+* CM automation actions:
+ * cm **test** report *( [See CM API](https://github.com/mlcommons/ck/tree/master/cm-mlops/automation/report/module.py#L15) )*
+
+
## ck
@@ -171,8 +204,9 @@ This file is generated automatically - don't edit!
* GitHub repository with CM automations: *cm pull [internal](https://github.com/mlcommons/ck/tree/master/cm/cmind/repo)*
* CM automation code and meta: *[GitHub](https://github.com/mlcommons/ck/tree/master/cm/cmind/repo/automation/automation)*
* CM automation actions:
- * cm **add** automation *( [See CM API](https://github.com/mlcommons/ck/tree/master/cm/cmind/repo/automation/automation/module.py#L15) )*
- * cm **doc** automation *( [See CM API](https://github.com/mlcommons/ck/tree/master/cm/cmind/repo/automation/automation/module.py#L87) )*
+ * cm **print_input** automation *( [See CM API](https://github.com/mlcommons/ck/tree/master/cm/cmind/repo/automation/automation/module.py#L15) )*
+ * cm **add** automation *( [See CM API](https://github.com/mlcommons/ck/tree/master/cm/cmind/repo/automation/automation/module.py#L39) )*
+ * cm **doc** automation *( [See CM API](https://github.com/mlcommons/ck/tree/master/cm/cmind/repo/automation/automation/module.py#L111) )*
diff --git a/docs/list_of_scripts.md b/docs/list_of_scripts.md
index 02bb816ba3..bb723bbb3f 100644
--- a/docs/list_of_scripts.md
+++ b/docs/list_of_scripts.md
@@ -23,7 +23,6 @@ design space exploration and deployment across continuously changing software an
* [Detection or installation of tools and artifacts](#detection-or-installation-of-tools-and-artifacts)
* [Docker automation](#docker-automation)
* [GUI](#gui)
-* [IPOL reproducibility](#ipol-reproducibility)
* [Legacy CK support](#legacy-ck-support)
* [ML/AI datasets](#mlai-datasets)
* [ML/AI frameworks](#mlai-frameworks)
@@ -35,6 +34,7 @@ design space exploration and deployment across continuously changing software an
* [Platform information](#platform-information)
* [Python automation](#python-automation)
* [Remote automation](#remote-automation)
+* [Reproduced papers](#reproduced-papers)
* [Reproducible papers](#reproducible-papers)
* [Tests](#tests)
* [TinyML automation](#tinyml-automation)
@@ -46,10 +46,48 @@ design space exploration and deployment across continuously changing software an
* [gui](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/gui)
+### Modular MLPerf benchmarks
+
+* [add-custom-nvidia-system](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/add-custom-nvidia-system)
+* [app-loadgen-generic-python](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/app-loadgen-generic-python)
+* [app-mlperf-inference](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/app-mlperf-inference)
+* [app-mlperf-inference-cpp](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/app-mlperf-inference-cpp)
+* [app-mlperf-inference-nvidia](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/app-mlperf-inference-nvidia)
+* [app-mlperf-inference-reference](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/app-mlperf-inference-reference)
+* [app-mlperf-inference-tflite-cpp](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/app-mlperf-inference-tflite-cpp)
+* [build-mlperf-inference-server-nvidia](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/build-mlperf-inference-server-nvidia)
+* [generate-mlperf-inference-submission](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/generate-mlperf-inference-submission)
+* [generate-mlperf-inference-user-conf](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/generate-mlperf-inference-user-conf)
+* [generate-mlperf-tiny-submission](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/generate-mlperf-tiny-submission)
+* [generate-nvidia-engine](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/generate-nvidia-engine)
+* [get-dataset-kits19](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/get-dataset-kits19)
+* [get-git-repo](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/get-git-repo)
+* [get-mlperf-inference-loadgen](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/get-mlperf-inference-loadgen)
+* [get-mlperf-inference-results](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/get-mlperf-inference-results)
+* [get-mlperf-inference-src](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/get-mlperf-inference-src)
+* [get-mlperf-inference-sut-configs](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/get-mlperf-inference-sut-configs)
+* [get-mlperf-inference-sut-description](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/get-mlperf-inference-sut-description)
+* [get-mlperf-power-dev](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/get-mlperf-power-dev)
+* [get-mlperf-training-src](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/get-mlperf-training-src)
+* [get-spec-ptd](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/get-spec-ptd)
+* [import-mlperf-inference-to-experiment](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/import-mlperf-inference-to-experiment)
+* [import-mlperf-tiny-to-experiment](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/import-mlperf-tiny-to-experiment)
+* [process-mlperf-accuracy](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/process-mlperf-accuracy)
+* [push-mlperf-inference-results-to-github](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/push-mlperf-inference-results-to-github)
+* [reproduce-mlperf-inference-nvidia](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/reproduce-mlperf-inference-nvidia)
+* [reproduce-mlperf-octoml-tinyml-results](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/reproduce-mlperf-octoml-tinyml-results)
+* [run-mlperf-inference-app](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/run-mlperf-inference-app)
+* [run-mlperf-inference-submission-checker](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/run-mlperf-inference-submission-checker)
+* [run-mlperf-power-client](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/run-mlperf-power-client)
+* [run-mlperf-power-server](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/run-mlperf-power-server)
+* [test-mlperf-inference-retinanet-win](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/test-mlperf-inference-retinanet-win)
+* [truncate-mlperf-inference-accuracy-log](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/truncate-mlperf-inference-accuracy-log)
+* [wrapper-reproduce-octoml-tinyml-submission](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/wrapper-reproduce-octoml-tinyml-submission)
+
### Modular ML/AI applications
-* [app-image-classification-onnx-cpp](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/app-image-classification-onnx-cpp)
* [app-image-classification-onnx-py](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/app-image-classification-onnx-py)
+* [app-image-classification-tf-onnx-cpp](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/app-image-classification-tf-onnx-cpp)
* [app-image-classification-torch-py](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/app-image-classification-torch-py)
* [app-image-classification-tvm-onnx-py](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/app-image-classification-tvm-onnx-py)
@@ -142,6 +180,7 @@ design space exploration and deployment across continuously changing software an
* [destroy-terraform](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/destroy-terraform)
* [get-aws-cli](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/get-aws-cli)
+* [get-rclone](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/get-rclone)
* [get-terraform](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/get-terraform)
* [install-aws-cli](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/install-aws-cli)
* [install-terraform-from-src](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/install-terraform-from-src)
@@ -185,42 +224,6 @@ design space exploration and deployment across continuously changing software an
* [benchmark-program](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/benchmark-program)
* [compile-program](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/compile-program)
-### Modular MLPerf benchmarks
-
-* [add-custom-nvidia-system](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/add-custom-nvidia-system)
-* [app-loadgen-generic-python](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/app-loadgen-generic-python)
-* [app-mlperf-inference](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/app-mlperf-inference)
-* [app-mlperf-inference-cpp](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/app-mlperf-inference-cpp)
-* [app-mlperf-inference-nvidia](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/app-mlperf-inference-nvidia)
-* [app-mlperf-inference-reference](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/app-mlperf-inference-reference)
-* [app-mlperf-inference-tflite-cpp](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/app-mlperf-inference-tflite-cpp)
-* [build-mlperf-inference-server-nvidia](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/build-mlperf-inference-server-nvidia)
-* [generate-mlperf-inference-submission](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/generate-mlperf-inference-submission)
-* [generate-mlperf-inference-user-conf](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/generate-mlperf-inference-user-conf)
-* [generate-mlperf-tiny-submission](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/generate-mlperf-tiny-submission)
-* [generate-nvidia-engine](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/generate-nvidia-engine)
-* [get-dataset-kits19](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/get-dataset-kits19)
-* [get-git-repo](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/get-git-repo)
-* [get-mlperf-inference-loadgen](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/get-mlperf-inference-loadgen)
-* [get-mlperf-inference-results](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/get-mlperf-inference-results)
-* [get-mlperf-inference-src](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/get-mlperf-inference-src)
-* [get-mlperf-inference-sut-configs](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/get-mlperf-inference-sut-configs)
-* [get-mlperf-inference-sut-description](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/get-mlperf-inference-sut-description)
-* [get-mlperf-power-dev](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/get-mlperf-power-dev)
-* [get-mlperf-training-src](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/get-mlperf-training-src)
-* [get-spec-ptd](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/get-spec-ptd)
-* [import-mlperf-inference-to-experiment](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/import-mlperf-inference-to-experiment)
-* [process-mlperf-accuracy](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/process-mlperf-accuracy)
-* [push-mlperf-inference-results-to-github](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/push-mlperf-inference-results-to-github)
-* [reproduce-mlperf-inference-nvidia](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/reproduce-mlperf-inference-nvidia)
-* [reproduce-mlperf-octoml-tinyml-results](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/reproduce-mlperf-octoml-tinyml-results)
-* [run-mlperf-inference-app](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/run-mlperf-inference-app)
-* [run-mlperf-inference-submission-checker](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/run-mlperf-inference-submission-checker)
-* [run-mlperf-power-client](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/run-mlperf-power-client)
-* [run-mlperf-power-server](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/run-mlperf-power-server)
-* [truncate-mlperf-inference-accuracy-log](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/truncate-mlperf-inference-accuracy-log)
-* [wrapper-reproduce-octoml-tinyml-submission](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/wrapper-reproduce-octoml-tinyml-submission)
-
### Dashboards
* [publish-results-to-dashboard](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/publish-results-to-dashboard)
@@ -247,7 +250,7 @@ design space exploration and deployment across continuously changing software an
* [print-python-version](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/print-python-version)
* [test-set-sys-user-cm](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/test-set-sys-user-cm)
-### IPOL reproducibility
+### Reproduced papers
* [app-ipol-reproducibility-2022-439](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/app-ipol-reproducibility-2022-439)
@@ -260,8 +263,8 @@ design space exploration and deployment across continuously changing software an
* [activate-python-venv](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/activate-python-venv) *(Activate virtual Python environment)*
* [add-custom-nvidia-system](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/add-custom-nvidia-system)
-* [app-image-classification-onnx-cpp](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/app-image-classification-onnx-cpp)
* [app-image-classification-onnx-py](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/app-image-classification-onnx-py)
+* [app-image-classification-tf-onnx-cpp](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/app-image-classification-tf-onnx-cpp)
* [app-image-classification-torch-py](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/app-image-classification-torch-py)
* [app-image-classification-tvm-onnx-py](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/app-image-classification-tvm-onnx-py)
* [app-image-corner-detection](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/app-image-corner-detection)
@@ -281,14 +284,21 @@ design space exploration and deployment across continuously changing software an
* [destroy-terraform](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/destroy-terraform)
* [detect-cpu](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/detect-cpu)
* [detect-os](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/detect-os)
+* [download-and-extract](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/download-and-extract)
+* [download-file](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/download-file)
+* [download-torrent](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/download-torrent)
+* [extract-file](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/extract-file)
* [flash-tinyml-binary](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/flash-tinyml-binary)
* [generate-mlperf-inference-submission](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/generate-mlperf-inference-submission)
* [generate-mlperf-inference-user-conf](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/generate-mlperf-inference-user-conf)
* [generate-mlperf-tiny-submission](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/generate-mlperf-tiny-submission)
* [generate-nvidia-engine](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/generate-nvidia-engine)
* [get-android-sdk](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/get-android-sdk)
+* [get-aocl](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/get-aocl)
* [get-aws-cli](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/get-aws-cli)
* [get-bazel](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/get-bazel)
+* [get-blis](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/get-blis)
+* [get-brew](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/get-brew)
* [get-ck](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/get-ck)
* [get-ck-repo-mlops](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/get-ck-repo-mlops)
* [get-cl](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/get-cl) *(Detect or install Microsoft C compiler)*
@@ -301,6 +311,7 @@ design space exploration and deployment across continuously changing software an
* [get-dataset-criteo](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/get-dataset-criteo)
* [get-dataset-imagenet-aux](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/get-dataset-imagenet-aux)
* [get-dataset-imagenet-helper](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/get-dataset-imagenet-helper)
+* [get-dataset-imagenet-train](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/get-dataset-imagenet-train)
* [get-dataset-imagenet-val](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/get-dataset-imagenet-val)
* [get-dataset-kits19](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/get-dataset-kits19)
* [get-dataset-librispeech](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/get-dataset-librispeech)
@@ -315,6 +326,7 @@ design space exploration and deployment across continuously changing software an
* [get-git-repo](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/get-git-repo)
* [get-github-cli](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/get-github-cli)
* [get-go](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/get-go)
+* [get-google-test](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/get-google-test)
* [get-ipol-src](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/get-ipol-src)
* [get-java](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/get-java)
* [get-javac](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/get-javac)
@@ -353,16 +365,21 @@ design space exploration and deployment across continuously changing software an
* [get-preprocessed-dataset-openimages](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/get-preprocessed-dataset-openimages)
* [get-preprocesser-script-generic](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/get-preprocesser-script-generic)
* [get-python3](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/get-python3)
+* [get-qaic-compute-sdk](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/get-qaic-compute-sdk)
+* [get-qaic-software-kit](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/get-qaic-software-kit)
+* [get-rclone](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/get-rclone)
* [get-spec-ptd](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/get-spec-ptd)
* [get-sys-utils-cm](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/get-sys-utils-cm)
* [get-sys-utils-min](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/get-sys-utils-min)
* [get-tensorrt](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/get-tensorrt)
* [get-terraform](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/get-terraform)
* [get-tvm](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/get-tvm)
+* [get-zendnn](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/get-zendnn)
* [get-zephyr](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/get-zephyr)
* [get-zephyr-sdk](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/get-zephyr-sdk)
* [gui](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/gui)
* [import-mlperf-inference-to-experiment](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/import-mlperf-inference-to-experiment)
+* [import-mlperf-tiny-to-experiment](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/import-mlperf-tiny-to-experiment)
* [install-aws-cli](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/install-aws-cli)
* [install-bazel](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/install-bazel)
* [install-cmake-prebuilt](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/install-cmake-prebuilt)
@@ -401,6 +418,7 @@ design space exploration and deployment across continuously changing software an
* [run-terraform](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/run-terraform)
* [set-echo-off-win](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/set-echo-off-win)
* [tar-my-folder](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/tar-my-folder)
+* [test-mlperf-inference-retinanet-win](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/test-mlperf-inference-retinanet-win)
* [test-set-sys-user-cm](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/test-set-sys-user-cm)
* [truncate-mlperf-inference-accuracy-log](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/truncate-mlperf-inference-accuracy-log)
* [wrapper-reproduce-octoml-tinyml-submission](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/wrapper-reproduce-octoml-tinyml-submission)
diff --git a/docs/taskforce.md b/docs/taskforce.md
index 00f5425d1b..089ecd5f1f 100644
--- a/docs/taskforce.md
+++ b/docs/taskforce.md
@@ -4,9 +4,13 @@
## Goals
-* Automatically run any MLPerf benchmark out-of-the-box with any software, hardware, model and data from any vendor: (prototype is available and validated during MLPerf inf v3.0 submission)
-* Automate optimization experiments and visualization of results with derived metrics: [prototype](https://cknowledge.org/mlcommons-inference-gui)
-* Generate Pareto-optimal end-to-end applications based on reproducible MLPerf results: (under development)
+* Automatically run any MLPerf benchmark out-of-the-box with any software, hardware, model and data from any vendor:
+ *[prototype is available and validated during MLPerf inf v3.0 submission](../cm-mlops/challenge/optimize-mlperf-inference-v3.1-2023/README.md)*.
+* Automate optimization experiments and visualization of results with derived metrics: *[prototype](https://cknowledge.org/mlcommons-inference-gui)*.
+* Generate Pareto-optimal end-to-end applications based on reproducible MLPerf results: *under development*.
+* Organize reproducibility, replicability and optimization challenges to improve MLPerf results across diverse software, hardware, models and data:
+ *[on-going](https://github.com/mlcommons/ck/tree/master/cm-mlops/challenge)
+ (see adopted terminology [here](artifact-evaluation/faq.md#what-is-the-difference-between-repeatability-reproducibility-and-replicability))*.
## Mission
@@ -20,7 +24,7 @@ Our goal is to help the community benchmark, optimize and compare AI, ML and oth
across diverse and rapidly evolving software, hardware, models and data from different vendors
in a unified, collaborative, automated and reproducible way using the common CM language.
-We are also organizing [reproducibility and optimization challenges](https://access.cknowledge.org/playground/?action=challenges)
+We are also organizing [reproducibility, replicability and optimization challenges](https://access.cknowledge.org/playground/?action=challenges)
powered by the common CM language and CK playground to reproduce results from research papers and MLPerf submissions,
optimize them in terms of performance, power consumption, accuracy, size, costs and other metrics,
and validate them in the real-world applications.
@@ -42,7 +46,7 @@ environment and data while slashing their development and operational costs.
## Discussions
-* Check our upcoming [reproducibility and optimization challenges](https://access.cknowledge.org/playground/?action=challenges)
+* Check our upcoming [reproducibility, replicability and optimization challenges](https://access.cknowledge.org/playground/?action=challenges)
* Join our [public Discord server](https://discord.gg/JjWNWXKxwT) to discuss developments and challenges.
* Join our [public conf-calls](https://docs.google.com/document/d/1zMNK1m_LhWm6jimZK6YE05hu4VH9usdbKJ3nBy-ZPAw).
* Check our [news](news.md).
@@ -99,6 +103,10 @@ environment and data while slashing their development and operational costs.
* [MLPerf universal C++ inference template library](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/app-mlperf-inference-cpp)
* [Portable, reusable and customizable CM scripts for technology-agnostic AI/ML/SW/HW co-design (Apache 2.0 license)](https://github.com/mlcommons/ck/tree/master/cm-mlops/script)
+* Reproducibility and replicability studies:
+ * [Terminology (ACM/NISO): Repeatability, Reproducibility and Replicability](artifact-evaluation/faq.md#what-is-the-difference-between-repeatability-reproducibility-and-replicability)
+ * [Artifact Evaluation at ML and systems conferences](https://cTuning.org/ae)
+
### Acknowledgments
This task force is supported by [MLCommons](https://mlcommons.org), [cTuning foundation](https://cTuning.org),