Skip to content

Latest commit

 

History

History
 
 

platform

Folders and files

NameName
Last commit message
Last commit date

parent directory

..
 
 
 
 
 
 
 
 

Collective Knowledge Playground

Introduction

The Collective Knowledge Playground (CK) is a free, open-source, and technology-agnostic on-prem platform being developed by the MLCommons task force on automation and reproducibility. It is intended to connect academia and industry to benchmark, optimize and compare AI, ML and other emerging applications across diverse and rapidly evolving models, software, hardware and data from different vendors in terms of costs, performance, power consumption, accuracy, size and other metrics in a unified, collaborative, automated, and reproducible way.

This platform is powered by the portable and technology-agnostic Collective Mind scripting language (MLCommons CM) with portable and reusable CM scripts developed by the community to solve the "AI/ML dependency hell". CM scripts help to automatically connect diverse and continuously changing models, software, hardware, data sets, best practices and optimization techniques into end-to-end applications in a transparent and non-intrusive way.

We thank the community for helping us to validate a prototype of the MLCommons CK playground by running and reproducing MLPerf inference v3.0 benchmarks: CK has helped to automatically interconnect very diverse technology from Neural Magic, Qualcomm, Krai, cKnowledge, OctoML, Deelvin, DELL, HPE, Lenovo, Hugging Face, Nvidia and Apple and run it across diverse CPUs, GPUs and DSPs with PyTorch, ONNX, QAIC, TF/TFLite, TVM and TensorRT using popular cloud providers (GCP, AWS, Azure) and individual servers and edge devices via our recent open optimization challenge.

Public GUI

Collaborative development

This open-source technology is being developed by the open MLCommons task force on automation and reproducibility led by Grigori Fursin and Arjun Suresh:

Source code for on-prem use

This platform is implemented as a portable automation recipe using the MLCommons CM scripting language.

Public challenges

Discuss your challenge in Discord, add your challenge here and create a PR.

Private challenges

You can use this platform to organize private challenges between your internal teams and external partners.

Install the MLCommons CK2 (CM) framework as described here.

Pull CM repository with portable MLOps automation recipes from the community:

cm pull repo mlcommons@ck

Run CK playground GUI on your local machine to aggregate, visualize and reproduce experiments:

cm run script "gui _playground" 

Check this script If you want to run the CK playground as a public or private server to run optimization experiments with your colleagues, external teams and users.

Copyright

2021-2023 MLCommons

License

Apache 2.0

Acknowledgments

This project is currently supported by MLCommons, cTuning foundation, cKnowledge and individual contributors. We thank HiPEAC and OctoML for sponsoring initial development.