Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

HetCCL: Accelerating LLM Training with Heterogeneous GPUs

Heehoon Kim, Jaehwan Lee, Taejeoung Kim, Jongwon Park, Jinpyo Kim, Pyongwon Suh, Ryan H. Choi, Sangwoo Lee, Jaejin Lee

TL;DR

HetCCL tackles the lack of cross-vendor collective communication in DL by enabling transparent interoperability between NVIDIA and AMD GPUs. It decouples cross-vendor coordination from vendor-specific kernels through a runtime API abstraction (TACC) and platform-specific, multi-vendor kernel compilation, while leveraging RDMA for efficient data movement. Key contributions include a cross-vendor RDMA mechanism, the TACC abstraction, zero-code modification deployment, and GPU-aware workload balancing, all validated on a multi-node, mixed-vendor cluster with end-to-end training speedups and preserved model accuracy. The work has practical impact by expanding hardware options for large-scale LLM training, reducing time-to-solution and enabling cost-effective utilization of heterogeneous GPU pools without altering existing DL frameworks.

Abstract

The rapid growth of large language models is driving organizations to expand their GPU clusters, often with GPUs from multiple vendors. However, current deep learning frameworks lack support for collective communication across heterogeneous GPUs, leading to inefficiency and higher costs. We present HetCCL, a collective communication library that unifies vendor-specific backends and enables RDMA-based communication across GPUs without requiring driver modifications. HetCCL introduces two novel mechanisms that enable cross-vendor communication while leveraging optimized vendor libraries, NVIDIA NCCL and AMD RCCL. Evaluations on a multi-vendor GPU cluster show that HetCCL matches NCCL and RCCL performance in homogeneous setups while uniquely scaling in heterogeneous environments, enabling practical, high-performance training with both NVIDIA and AMD GPUs without changes to existing deep learning applications.

HetCCL: Accelerating LLM Training with Heterogeneous GPUs

TL;DR

HetCCL tackles the lack of cross-vendor collective communication in DL by enabling transparent interoperability between NVIDIA and AMD GPUs. It decouples cross-vendor coordination from vendor-specific kernels through a runtime API abstraction (TACC) and platform-specific, multi-vendor kernel compilation, while leveraging RDMA for efficient data movement. Key contributions include a cross-vendor RDMA mechanism, the TACC abstraction, zero-code modification deployment, and GPU-aware workload balancing, all validated on a multi-node, mixed-vendor cluster with end-to-end training speedups and preserved model accuracy. The work has practical impact by expanding hardware options for large-scale LLM training, reducing time-to-solution and enabling cost-effective utilization of heterogeneous GPU pools without altering existing DL frameworks.

Abstract

The rapid growth of large language models is driving organizations to expand their GPU clusters, often with GPUs from multiple vendors. However, current deep learning frameworks lack support for collective communication across heterogeneous GPUs, leading to inefficiency and higher costs. We present HetCCL, a collective communication library that unifies vendor-specific backends and enables RDMA-based communication across GPUs without requiring driver modifications. HetCCL introduces two novel mechanisms that enable cross-vendor communication while leveraging optimized vendor libraries, NVIDIA NCCL and AMD RCCL. Evaluations on a multi-vendor GPU cluster show that HetCCL matches NCCL and RCCL performance in homogeneous setups while uniquely scaling in heterogeneous environments, enabling practical, high-performance training with both NVIDIA and AMD GPUs without changes to existing deep learning applications.
Paper Structure (60 sections, 16 figures, 4 tables)

This paper contains 60 sections, 16 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Background and Related Work
  3. Communication Mechanisms of GPUs
  4. Intra-node communication.
  5. Inter-node communication and RDMA.
  6. Existing CCLs for GPUs
  7. Vendor-provided libraries.
  8. Other libraries.
  9. Motivation and Challenges
  10. Motivation
  11. Challenges
  12. Near-native performance in homogeneous settings.
  13. High performance in heterogeneous settings.
  14. Heterogeneity in software stacks across vendors.
  15. Performance disparity and stragglers in heterogeneous collectives.
  16. ...and 45 more sections

Figures (16)

  • Figure 1: Communication between GPUs across nodes.
  • Figure 2: Comparison between the previous approach and HetCCL.
  • Figure 3: RDMA-based point-to-point communication between heterogeneous GPUs in different nodes. Each side allocates buffers via its own GPU runtime and uses IB Verbs for communication.
  • Figure 4: Runtime API abstraction layer in HetCCL.
  • Figure 5: Device code compilation mechanism in HetCCL.
  • ...and 11 more figures