Organization:MLCommons

From HandWiki

The MLCommons Association is a global and open non-profit organization dedicated to making machine learning (ML) better for everyone. MLCommons is an industry and academic consortium with over 100 members that focuses on collective engineering for machine learning and artificial intelligence. MLCommons initiatives fall under four pillars: benchmarks and metrics, data sets, best practices, and research. MLCommons is an Open source focused organization and releases software using the Apache License.

MLCommons
Mlcommons logo spelled out dark background.png
Formation2020
Type501(c)(6) organization
PurposeMaking machine learning better for everyone
HeadquartersSan Francisco, California
Membership
100+ corporations, researchers, individuals, and other entities
Executive Director
David Kanter
Websitewww.mlcommons.org

Benchmarks

The MLPerf benchmark suites are the industry-standard for measuring machine learning system performance. MLPerf started as an informal collaboration between researchers and industry and was inspired by industry-standard benchmarks for computer performance and prior efforts in ML benchmarking[1][2] and is now used by a diverse group of researchers.[3][4]

MLPerf includes a number of different benchmark suites focused on different types of systems and workloads:

  • HPC - Measures the speed of training ML models that solve research problems on large scientific datasets using supercomputers. Examples tasks include climate simulation and prediction and cosmological simulation.[5]
  • Training - Measures the speed of training ML models on that solve commercial problems on large datasets. Examples tasks include recommendation, natural language processing, speech-to-text, object detection, and image recognition.[6]
  • Inference - Measures how fast datacenter and edge systems process inputs and produce results using a trained model. Optionally includes measuring power measurement. Examples tasks include recommendation, question answering, speech-to-text, object detection, and image recognition.[7]
  • Mobile - Measures how fast smartphone, tablet, and notebook systems process inputs and produce results using a trained model. Example tasks include image segmentation, object detection, and question answering.[8]
  • Tiny - Measures how fast Internet of things and low-power embedded systems process inputs and produce results using a trained model. Optionally includes measuring power measurement. Examples tasks include key word spotting, visual wake-word detection, and anomaly detection.[9]

Each MLPerf benchmark suite is typically released on a 6-month cycle with updates to the workloads and new results.

Data sets

MLCommons builds and maintains large open and diverse data sets that are released under the Creative Commons CC-BY license.

  • People's Speech - The People’s Speech is a free-to-download 31,400-hour and growing supervised conversational English speech recognition dataset licensed for academic and commercial usage under CC-BY.[10]
  • Multi-lingual Spoken Words Corpus - Multilingual Spoken Words Corpus is a large and growing audio dataset of spoken words in 50 languages collectively spoken by over 5 billion people, for academic research and commercial applications in keyword spotting and spoken term search, licensed under CC-BY 4.0. The dataset contains more than 340,000 keywords, totaling 23.4 million 1-second spoken examples (over 6,000 hours).[11]

Best Practices

MLCommons creates best practices that make using machine easier and accessible to more people and companies.

  • MLCube - MLCube brings the concept of interchangeable parts to the world of machine learning models. It is the shipping container that enables researchers and developers to easily share the software that powers machine learning. MLCube is a set of common conventions for creating ML software that can just "plug-and-play" on many systems. MLCube makes it easier for researchers to share innovative ML models, for a developer to experiment with many models, and for software companies to create infrastructure for models.

Membership

MLCommons is open to members and affiliates of all sizes. Large organizations (>500 people) and small organizations (>10 people) pay to become members or affiliates, while qualified academics, individuals, and very small organizations (<10 people) are admitted at no cost.

MLCommons comprises over 100 members across the world, including over 50 companies and researchers from over a dozen academic institutions.

History

MLCommons was formed and launched in 2020.[12] The foundation for MLCommons started with the MLPerf benchmarks in 2018, which established industry-standard metrics to measure machine learning performance and quickly grew to encompass data sets and best practices. The MLPerf benchmarks played a critical role for industry and research and were tremendously popular. The community quickly spread across nearly every continent and grew to over 70 supporting organizations from software startups, to researchers at top universities, and to cloud computing and semiconductor giants. MLCommons was formed to steward the MLPerf benchmarks, data sets, and best practices.

References

  1. ↑ Patterson, David; Diamos, Greg; Young, Cliff; Mattson, Peter; Bailis, Peter; Wei, Gu-Yeon. "MLPerf: A benchmark suite for machine learning from an academic-industry cooperative: Artificial Intelligence Conference: AI & machine learning training" (in en). https://conferences.oreilly.com/artificial-intelligence/ai-ny-2018/public/schedule/detail/69826.html. 
  2. ↑ Kanter, David (2019-03-20). "Why I joined MLPerf". https://www.eetimes.com/why-i-joined-mlperf/. 
  3. ↑ Kumar, Sameer; Wang, Yu Emma; Young, Cliff; Bradbury, James; Levskaya, Anselm; Hechtman, Blake; Chen, Dehao; Lee, Hyouk Joong et al. (2020). "Exploring the Limits of Concurrency in ML Training on Google TPUs: Proceedings of the Third Conference on Machine Learning and Systems (MLSys 2021)" (in en). https://proceedings.mlsys.org/paper/2021/file/28dd2c7955ce926456240b2ff0100bde-Paper.pdf. 
  4. ↑ Moore, Samuel K. (2021-09-24). "Benchmark Shows AIs Are Getting Speedier" (in en). IEEE Spectrum. https://spectrum.ieee.org/ai-benchmarks-mlperf-performance. 
  5. ↑ "Supercomputers Flex Their AI Muscles". 20 November 2021. https://spectrum.ieee.org/ai-supercomputer. 
  6. ↑ "Is AI Training Outstripping Moore's Law?". 2 December 2021. https://spectrum.ieee.org/ai-training-mlperf. 
  7. ↑ Reddi, Vijay Janapa; Cheng, Christine; Kanter, David et al. (2020). "MLPerf Inference Benchmark". 2020 ACM/IEEE 47th Annual International Symposium on Computer Architecture (ISCA): 446–459. doi:10.1109/ISCA45697.2020.00045. ISBN 978-1-7281-4661-4. 
  8. ↑ Reddi, Vijay Janapa; Kanter, David; Mattson, Peter; et al. (2021-02-26). "MLPerf Mobile Inference Benchmark". arXiv:2012.02328 [cs.LG].
  9. ↑ Banbury, Colby; Reddi, Vijay Janapa; Torelli, Peter; et al. (2021-08-24). "MLPerf Tiny Benchmark". arXiv:2106.07597 [cs.LG].
  10. ↑ Galvez, Daniel; Diamos, Greg; Torres, Juan Manuel Ciro et al. (2021-06-08) (in en). The People's Speech: A Large-Scale Diverse English Speech Recognition Dataset for Commercial Usage. https://openreview.net/forum?id=R8CwidgJ0yT. 
  11. ↑ Mazumder, Mark; Chitlangia, Sharad; Banbury, Colby et al. (2021-08-21) (in en). Multilingual Spoken Words Corpus. https://openreview.net/forum?id=c20jiJ5K2H. 
  12. ↑ "MLCommons™ Launches" (in en). https://mlcommons.org/. 




Retrieved from "https://handwiki.org/wiki/index.php?title=Organization:MLCommons&oldid=3394233"