Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Introducing Milabench: Benchmarking Accelerators for AI

Pierre Delaunay, Xavier Bouthillier, Olivier Breuleux, Satya Ortiz-Gagné, Olexa Bilaniuk, Fabrice Normandin, Arnaud Bergeron, Bruno Carrez, Guillaume Alain, Soline Blanc, Frédéric Osterrath, Joseph Viviano, Roger Creus-Castanyer Darshan Patil, Rabiul Awal, Le Zhang

TL;DR

This report introduces Milabench, the resulting benchmarking suite, and details the design methodology, the structure of the benchmarking suite, and provides performance evaluations using GPUs from NVIDIA, AMD, and Intel.

Abstract

AI workloads, particularly those driven by deep learning, are introducing novel usage patterns to high-performance computing (HPC) systems that are not comprehensively captured by standard HPC benchmarks. As one of the largest academic research centers dedicated to deep learning, Mila identified the need to develop a custom benchmarking suite to address the diverse requirements of its community, which consists of over 1,000 researchers. This report introduces Milabench, the resulting benchmarking suite. Its design was informed by an extensive literature review encompassing 867 papers, as well as surveys conducted with Mila researchers. This rigorous process led to the selection of 26 primary benchmarks tailored for procurement evaluations, alongside 16 optional benchmarks for in-depth analysis. We detail the design methodology, the structure of the benchmarking suite, and provide performance evaluations using GPUs from NVIDIA, AMD, and Intel. The Milabench suite is open source and can be accessed at github.com/mila-iqia/milabench.

Introducing Milabench: Benchmarking Accelerators for AI

TL;DR

This report introduces Milabench, the resulting benchmarking suite, and details the design methodology, the structure of the benchmarking suite, and provides performance evaluations using GPUs from NVIDIA, AMD, and Intel.

Abstract

AI workloads, particularly those driven by deep learning, are introducing novel usage patterns to high-performance computing (HPC) systems that are not comprehensively captured by standard HPC benchmarks. As one of the largest academic research centers dedicated to deep learning, Mila identified the need to develop a custom benchmarking suite to address the diverse requirements of its community, which consists of over 1,000 researchers. This report introduces Milabench, the resulting benchmarking suite. Its design was informed by an extensive literature review encompassing 867 papers, as well as surveys conducted with Mila researchers. This rigorous process led to the selection of 26 primary benchmarks tailored for procurement evaluations, alongside 16 optional benchmarks for in-depth analysis. We detail the design methodology, the structure of the benchmarking suite, and provide performance evaluations using GPUs from NVIDIA, AMD, and Intel. The Milabench suite is open source and can be accessed at github.com/mila-iqia/milabench.

Paper Structure

This paper contains 16 sections, 1 equation, 3 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Suite Design
  3. Literature Review
  4. Surveys
  5. Selection of benchmarks
  6. Implementation
  7. Benchmarking Methodology
  8. Results from Milabench
  9. Optional Benchmarks
  10. Main benchmarks
  11. Related Work
  12. Future Directions for Milabench
  13. Conclusion
  14. Hardware & Configuration
  15. GPT-4o Response Example
  16. ...and 1 more sections

Figures (3)

  • Figure 1: Performance comparison between different data types
  • Figure 2: Results Overview
  • Figure 3: Design breakdown across 5 different major characteristics. The beige columns represent the benchmarks included in the main suite. The height of the cells represents their weight (ex: dimenet has a weight of 2 while ppo has a weight of 1). The blue bars are the targets we determined based on our literature review and internal surveys. Each benchmark appears at least once in all of these tables. The columns are not mutually exclusive in all tables except for model sizes. To design Milabench's suite, we selected benchmarks such that the proportions of the benchmarks in each column of these tables are as close as possible to the targets.