Click here to see the table of contents.

• Description
• Information
• Usage
  • CM installation
  • CM script automation help
  • CM CLI
  • CM Python API
  • CM GUI
  • CM modular Docker container
• Customization
  • Default environment
• Script workflow, dependencies and native scripts
• Script output
• New environment keys (filter)
• New environment keys auto-detected from customize
• Maintainers

Note that this README is automatically generated - don't edit! See more info.

Description

See more info.

Information

• Category: Modular MLPerf benchmarks.
• CM GitHub repository: mlcommons@ck
• GitHub directory for this script: GitHub
• CM meta description for this script: _cm.json
• CM "database" tags to find this script: generate,submission,mlperf,mlperf-tiny,tiny,mlcommons,tiny-submission,mlperf-tiny-submission,mlcommons-tiny-submission
• Output cached?: False

---

Usage

CM installation

Guide

CM pull repository

cm pull repo mlcommons@ck

CM script automation help

cm run script --help

CM CLI

1. cm run script --tags=generate,submission,mlperf,mlperf-tiny,tiny,mlcommons,tiny-submission,mlperf-tiny-submission,mlcommons-tiny-submission

2. cm run script "generate submission mlperf mlperf-tiny tiny mlcommons tiny-submission mlperf-tiny-submission mlcommons-tiny-submission"

3. cm run script 04289b9fc07b42b6

• variations can be seen here

• input_flags can be seen here

CM Python API

Click here to expand this section.

import cmind

r = cmind.access({'action':'run'
                  'automation':'script',
                  'tags':'generate,submission,mlperf,mlperf-tiny,tiny,mlcommons,tiny-submission,mlperf-tiny-submission,mlcommons-tiny-submission'
                  'out':'con',
                  ...
                  (other input keys for this script)
                  ...
                 })

if r['return']>0:
    print (r['error'])

CM GUI

cm run script --tags=gui --script="generate,submission,mlperf,mlperf-tiny,tiny,mlcommons,tiny-submission,mlperf-tiny-submission,mlcommons-tiny-submission"

Use this online GUI to generate CM CMD.

CM modular Docker container

TBD

---

Customization

Default environment

Click here to expand this section.

These keys can be updated via --env.KEY=VALUE or env dictionary in @input.json or using script flags.

---

Script workflow, dependencies and native scripts

Click here to expand this section.

1. Read "deps" on other CM scripts from meta
   • get,python3
     • CM names: --adr.['python', 'python3']...
     • CM script: get-python3
   • get,sut,system-description
     • CM script: get-mlperf-inference-sut-description
2. Run "preprocess" function from customize.py
3. Read "prehook_deps" on other CM scripts from meta
4. Run native script if exists
5. Read "posthook_deps" on other CM scripts from meta
6. Run "postrocess" function from customize.py
7. Read "post_deps" on other CM scripts from meta *
   • if (CM_MLPERF_RUN_STYLE == valid)
   • CM script: activate-python-venv
   • CM script: add-custom-nvidia-system
   • CM script: app-image-classification-onnx-py
   • CM script: app-image-classification-tf-onnx-cpp
   • CM script: app-image-classification-torch-py
   • CM script: app-image-classification-tvm-onnx-py
   • CM script: app-image-corner-detection
   • CM script: app-ipol-reproducibility-2022-439
   • CM script: app-loadgen-generic-python
   • CM script: app-mlperf-inference
   • CM script: app-mlperf-inference-cpp
   • CM script: app-mlperf-inference-nvidia
   • CM script: app-mlperf-inference-reference
   • CM script: app-mlperf-inference-tflite-cpp
   • CM script: benchmark-program
   • CM script: build-docker-image
   • CM script: build-dockerfile
   • CM script: build-mlperf-inference-server-nvidia
   • CM script: compile-program
   • CM script: convert-ml-model-huggingface-to-onnx
   • CM script: destroy-terraform
   • CM script: detect-cpu
   • CM script: detect-os
   • CM script: download-and-extract
   • CM script: download-torrent
   • CM script: flash-tinyml-binary
   • CM script: generate-mlperf-inference-submission
   • CM script: generate-mlperf-inference-user-conf
   • CM script: generate-mlperf-tiny-submission
   • CM script: generate-nvidia-engine
   • CM script: get-android-sdk
   • CM script: get-aocl
   • CM script: get-aws-cli
   • CM script: get-bazel
   • CM script: get-blis
   • CM script: get-brew
   • CM script: get-ck
   • CM script: get-ck-repo-mlops
   • CM script: get-cl
   • CM script: get-cmake
   • CM script: get-cmsis_5
   • CM script: get-compiler-flags
   • CM script: get-cuda
   • CM script: get-cuda-devices
   • CM script: get-cudnn
   • CM script: get-dataset-criteo
   • CM script: get-dataset-imagenet-aux
   • CM script: get-dataset-imagenet-helper
   • CM script: get-dataset-imagenet-train
   • CM script: get-dataset-imagenet-val
   • CM script: get-dataset-kits19
   • CM script: get-dataset-librispeech
   • CM script: get-dataset-openimages
   • CM script: get-dataset-openimages-annotations
   • CM script: get-dataset-squad
   • CM script: get-dataset-squad-vocab
   • CM script: get-dlrm
   • CM script: get-gcc
   • CM script: get-generic-python-lib
   • CM script: get-generic-sys-util
   • CM script: get-git-repo
   • CM script: get-github-cli
   • CM script: get-go
   • CM script: get-google-test
   • CM script: get-ipol-src
   • CM script: get-java
   • CM script: get-javac
   • CM script: get-lib-armnn
   • CM script: get-lib-dnnl
   • CM script: get-llvm
   • CM script: get-microtvm
   • CM script: get-ml-model-3d-unet-kits19
   • CM script: get-ml-model-bert-base-squad
   • CM script: get-ml-model-bert-large-squad
   • CM script: get-ml-model-dlrm-terabyte
   • CM script: get-ml-model-efficientnet-lite
   • CM script: get-ml-model-huggingface-zoo
   • CM script: get-ml-model-mobilenet
   • CM script: get-ml-model-neuralmagic-zoo
   • CM script: get-ml-model-resnet50
   • CM script: get-ml-model-resnet50-tvm
   • CM script: get-ml-model-retinanet
   • CM script: get-ml-model-retinanet-nvidia
   • CM script: get-ml-model-rnnt
   • CM script: get-ml-model-using-imagenet-from-model-zoo
   • CM script: get-mlperf-inference-loadgen
   • CM script: get-mlperf-inference-nvidia-common-code
   • CM script: get-mlperf-inference-results
   • CM script: get-mlperf-inference-src
   • CM script: get-mlperf-inference-sut-configs
   • CM script: get-mlperf-inference-sut-description
   • CM script: get-mlperf-power-dev
   • CM script: get-mlperf-training-src
   • CM script: get-onnxruntime-prebuilt
   • CM script: get-openssl
   • CM script: get-preprocessed-dataset-criteo
   • CM script: get-preprocesser-script-generic
   • CM script: get-preprocessed-dataset-imagenet
   • CM script: get-preprocessed-dataset-kits19
   • CM script: get-preprocessed-dataset-librispeech
   • CM script: get-preprocessed-dataset-openimages
   • CM script: get-python3
   • CM script: get-qaic-compute-sdk
   • CM script: get-qaic-software-kit
   • CM script: get-rclone
   • CM script: get-spec-ptd
   • CM script: get-sys-utils-cm
   • CM script: get-sys-utils-min
   • CM script: get-tensorrt
   • CM script: get-terraform
   • CM script: get-tvm
   • CM script: get-zendnn
   • CM script: get-zephyr
   • CM script: get-zephyr-sdk
   • CM script: gui
   • CM script: import-mlperf-inference-to-experiment
   • CM script: import-mlperf-tiny-to-experiment
   • CM script: install-aws-cli
   • CM script: install-bazel
   • CM script: install-cmake-prebuilt
   • CM script: install-cuda-package-manager
   • CM script: install-cuda-prebuilt
   • CM script: install-gcc-src
   • CM script: install-gflags
   • CM script: install-github-cli
   • CM script: install-llvm-prebuilt
   • CM script: install-llvm-src
   • CM script: install-openssl
   • CM script: install-python-src
   • CM script: install-python-venv
   • CM script: install-tensorflow-for-c
   • CM script: install-tensorflow-from-src
   • CM script: install-terraform-from-src
   • CM script: install-tflite-from-src
   • CM script: print-hello-world
   • CM script: print-hello-world-java
   • CM script: print-hello-world-json
   • CM script: print-hello-world-py
   • CM script: print-python-version
   • CM script: process-mlperf-accuracy
   • CM script: publish-results-to-dashboard
   • CM script: push-csv-to-spreadsheet
   • CM script: push-mlperf-inference-results-to-github
   • CM script: remote-run-commands
   • CM script: reproduce-mlperf-inference-nvidia
   • CM script: reproduce-mlperf-octoml-tinyml-results
   • CM script: run-docker-container
   • CM script: run-mlperf-inference-app
   • CM script: run-mlperf-inference-mobilenet-models
   • CM script: run-mlperf-inference-submission-checker
   • CM script: run-mlperf-power-client
   • CM script: run-mlperf-power-server
   • CM script: run-terraform
   • CM script: set-echo-off-win
   • CM script: tar-my-folder
   • CM script: test-mlperf-inference-retinanet-win
   • CM script: test-set-sys-user-cm
   • CM script: truncate-mlperf-inference-accuracy-log
   • CM script: wrapper-reproduce-octoml-tinyml-submission
   • CM script: test-script1
   • CM script: test-script2
   • CM script: test-script3
   • CM script: test-script4
   • CM script: test-script5
   • CM script: app-generate-image-dalle-mini-jax-py
   • CM script: app-generate-image-stable-diffusion2-pytorch-cuda-py
   • CM script: app-image-classification-onnx-py-ck
   • CM script: app-image-corner-detection-old
   • CM script: app-ipol-demo
   • CM script: app-stable-diffusion-pytorch-cuda-py
   • CM script: not-needed--get-android-cmdline-tools
   • CM script: not-needed--install-android-cmdline-tools
   • CM script: process-mlperf-inference-results
   • CM script: download-file
   • CM script: extract-file

---

Script output

New environment keys (filter)

New environment keys auto-detected from customize

---

Maintainers

• Open MLCommons taskforce on automation and reproducibility