The Collective Knowledge Playground (CK) is a free, open-source, and technology-agnostic on-prem platform being developed by the MLCommons task force on automation and reproducibility. It is intended to connect academia and industry to benchmark, optimize and compare AI, ML and other emerging applications across diverse and rapidly evolving models, software, hardware and data from different vendors in terms of costs, performance, power consumption, accuracy, size and other metrics in a unified, collaborative, automated, and reproducible way.
This platform is powered by the portable and technology-agnostic Collective Mind scripting language (MLCommons CM) with portable and reusable CM scripts developed by the community to solve the "AI/ML dependency hell". CM scripts help to automatically connect diverse and continuously changing models, software, hardware, data sets, best practices and optimization techniques into end-to-end applications in a transparent and non-intrusive way.
We thank the community for helping us to validate a prototype of the MLCommons CK playground by running and reproducing MLPerf inference v3.0 benchmarks: CK has helped to automatically interconnect very diverse technology from Neural Magic, Qualcomm, Krai, cKnowledge, OctoML, Deelvin, DELL, HPE, Lenovo, Hugging Face, Nvidia and Apple and run it across diverse CPUs, GPUs and DSPs with PyTorch, ONNX, QAIC, TF/TFLite, TVM and TensorRT using popular cloud providers (GCP, AWS, Azure) and individual servers and edge devices via our recent open optimization challenge.
This open-source technology is being developed by the open MLCommons task force on automation and reproducibility led by Grigori Fursin and Arjun Suresh:
- Join our public Discord server.
- Join our public conf-calls.
- Check our news.
- Check our presentation with development plans.
- Read about our CK concept (previous version before MLCommons).
This platform is implemented as a portable automation recipe using the MLCommons CM scripting language.
Discuss your challenge in Discord, add your challenge here and create a PR.
You can use this platform to organize private challenges between your internal teams and external partners.
Install the MLCommons CK2 (CM) framework as described here.
Pull CM repository with portable MLOps automation recipes from the community:
cm pull repo mlcommons@ck
Run CK playground GUI on your local machine to aggregate, visualize and reproduce experiments:
cm run script "gui _playground"
Check this script If you want to run the CK playground as a public or private server to run optimization experiments with your colleagues, external teams and users.
2021-2023 MLCommons
This project is currently supported by MLCommons, cTuning foundation, cKnowledge and individual contributors. We thank HiPEAC and OctoML for sponsoring initial development.