Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Performance and Power: Systematic Evaluation of AI Workloads on Accelerators with CARAML

Chelsea Maria John, Stepan Nassyr, Carolin Penke, Andreas Herten

TL;DR

CARAML introduces an open, compact benchmark suite to quantify both performance and energy consumption of AI training workloads on heterogeneous accelerators, using LLM (GPT-style) and ResNet50 tasks. It combines two workload implementations (Megatron-LM-based LLM training on GPUs and TensorFlow-based ResNet50 on GPUs/TPUs with IPUs) with JUBE-driven automation and the jpwr energy-measurement tool to enable reproducible cross-hardware comparisons. The work provides detailed results across NVIDIA A100/H100, AMD MI250, GH200, and Graphcore GC200 platforms, highlighting throughput gains on newer hardware as well as nuanced energy-efficiency trends shaped by interconnect, memory bandwidth, and workload-parallelism choices. It also documents substantial technical challenges in software portability, containerization, and HPC system configurations, offering practical guidance for future cross-architecture benchmarking efforts and outlining directions for expanding workloads and continuous benchmarking.

Abstract

The rapid advancement of machine learning (ML) technologies has driven the development of specialized hardware accelerators designed to facilitate more efficient model training. This paper introduces the CARAML benchmark suite, which is employed to assess performance and energy consumption during the training of transformer-based large language models and computer vision models on a range of hardware accelerators, including systems from NVIDIA, AMD, and Graphcore. CARAML provides a compact, automated, extensible, and reproducible framework for assessing the performance and energy of ML workloads across various novel hardware architectures. The design and implementation of CARAML, along with a custom power measurement tool called jpwr, are discussed in detail.

Performance and Power: Systematic Evaluation of AI Workloads on Accelerators with CARAML

TL;DR

CARAML introduces an open, compact benchmark suite to quantify both performance and energy consumption of AI training workloads on heterogeneous accelerators, using LLM (GPT-style) and ResNet50 tasks. It combines two workload implementations (Megatron-LM-based LLM training on GPUs and TensorFlow-based ResNet50 on GPUs/TPUs with IPUs) with JUBE-driven automation and the jpwr energy-measurement tool to enable reproducible cross-hardware comparisons. The work provides detailed results across NVIDIA A100/H100, AMD MI250, GH200, and Graphcore GC200 platforms, highlighting throughput gains on newer hardware as well as nuanced energy-efficiency trends shaped by interconnect, memory bandwidth, and workload-parallelism choices. It also documents substantial technical challenges in software portability, containerization, and HPC system configurations, offering practical guidance for future cross-architecture benchmarking efforts and outlining directions for expanding workloads and continuous benchmarking.

Abstract

The rapid advancement of machine learning (ML) technologies has driven the development of specialized hardware accelerators designed to facilitate more efficient model training. This paper introduces the CARAML benchmark suite, which is employed to assess performance and energy consumption during the training of transformer-based large language models and computer vision models on a range of hardware accelerators, including systems from NVIDIA, AMD, and Graphcore. CARAML provides a compact, automated, extensible, and reproducible framework for assessing the performance and energy of ML workloads across various novel hardware architectures. The design and implementation of CARAML, along with a custom power measurement tool called jpwr, are discussed in detail.
Paper Structure (24 sections, 4 figures, 3 tables)

This paper contains 24 sections, 4 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Background
  3. Natural Language Processing
  4. Computer Vision
  5. Accelerators
  6. Related Work
  7. The CARAML benchmark
  8. Benchmark Details
  9. LLM Training
  10. ResNet50 Training
  11. Automation with JUBE
  12. Power Measurements with jpwr
  13. Benchmark Execution
  14. Results
  15. LLM Training
  16. ...and 9 more sections

Figures (4)

  • Figure 1: List of evaluated accelerators. NVIDIA's Streaming Multiprocessors are abbreviated with SM. AMD's Compute Units are abbreviated with CU. Peak performance is given without sparsity.
  • Figure 2: Throughput and energy efficiency for LLM training on NVIDIA and AMD systems using a 800M GPT model.
  • Figure 3: Throughput and energy consumption for ResNet50 model training on a single device of NVIDIA and AMD systems.
  • Figure 4: Throughput for ResNet50 training depending on number of GPUs and global batch size on various systems. OOM stands for Out of Memory, i.e. the batch size is too large for the memory of the device.