Machine Learning (ML) System Benchmark Task

A Machine Learning (ML) System Benchmark Task is an ML task that is an AI benchmarking task (that evaluates ML system performance).

• Context:
  • Task Input: ML Benchmark Datasets.
  • Task Output: ML System Performance Score.
  • Task Requirement:
    • Machine Learning Baseline Models.
    • a Machine Learning Model Training System.
    • a ML Benchmark Performance Metrics.
  • ...
• Example(s):
  • MLBench Benchmark Task,
  • DAWNBench Benchrmak Task,
  • MLPerf Benchmark Task,
  • Penn Machine Learning Benchmark (PMLB) Task,
  • MoleculeNet Benchmark Task.
  • …
• Counter-Example(s):
  • a Natural Language Processing System Benchmark Task,
  • a WikiText Error Correction (WTEC) System Benchmark Task,
  • a Text Error Correction System Benchmark Task,
  • a Optical Character Recognition (OCR) System Benchmark System,
  • a Data Mining System Benchmark Task,
  • a Word Processing Software Benchmark Task,
  • a Database Management Systems Benchmark Task.
• See: Learning Rate, Machine Learning Model, Machine Learning Algorithm, Evaluation System, Computer Hardware Benchmarking, Computing System Benchmarking.

---

---

References

2019

• (John & Mattson, 2019) ⇒ Tom St. John, and Peter Mattson (2019)."MLPerf: A Benchmark for Machine Learning". In: Proceedings of the International Conference for High Performance Computing, Networking, Storage, and Analysis (SC19).

2018a

• (Balaji & Allen, 2018) ⇒ Adithya Balaji, Alexander Allen (2018). "Benchmarking Automatic Machine Learning Frameworks". arXiv preprint arXiv:1808.06492.

2018b

• (Liu et al. 2018) ⇒ Yu Liu, Hantian Zhang, Luyuan Zeng, Wentao Wu, and Ce Zhang (2018). "MLbench: MLBench: Benchmarking Machine Learning Services Against Human Experts". Proceedings of the VLDB Endowment, 11(10), 1220-1232. DOI:10.14778/3231751.3231770

2018c

• (Wu et al., 2018) ⇒ Zhenqin Wu, Bharath Ramsundar, Evan N. Feinberg, Joseph Gomes, Caleb Geniesse, Aneesh S. Pappu, Karl Leswing, and Vijay Pande (2018). "MoleculeNet: A Benchmark For Molecular Machine Learning". Chemical science, 9(2), 513-530.

2017a

• (Olson et al., 2017) ⇒ Randal S. Olson, William La Cava, Patryk Orzechowski, Ryan J. Urbanowicz, and Jason H. Moore (2017). "PMLB: A Large Benchmark Suite For Machine Learning Evaluation And Comparison". BioData mining, 10, 36. DOI:10.1186/s13040-017-0154-4

2017b

• (Coleman et al., 2017) ⇒ Cody Coleman, Deepak Narayanan, Daniel Kang, Tian Zhao, Jian Zhang, Luigi Nardi, Peter Bailis, Kunle Olukotun, Chris Re, and Matei Zaharia (2017). "Dawnbench: An End-To-End Deep Learning Benchmark And Competition". Training, 100(101), 102.

---