Holmes: Towards Distributed Training Across Clusters with Heterogeneous NIC Environment

TL;DR

Holmes tackles the high cost of training large language models by enabling efficient distributed training across GPU clusters with heterogeneous NICs. It introduces NIC-aware scheduling and three key components — Cross-cluster Pipeline Parallelism, Automatic NIC Selection, and Self-Adapting Pipeline Partition — to align communication patterns with NIC capabilities. Empirical results show Holmes achieves performance close to homogeneous InfiniBand or RoCE networks and outperforms Ethernet-only setups and mainstream LLM frameworks in heterogeneous settings. The work democratizes large-scale LLM training by leveraging existing clusters without requiring uniform high-speed interconnects, with practical impact for research communities and industry alike.

Abstract

Large language models (LLMs) such as GPT-3, OPT, and LLaMA have demonstrated remarkable accuracy in a wide range of tasks. However, training these models can incur significant expenses, often requiring tens of thousands of GPUs for months of continuous operation. Typically, this training is carried out in specialized GPU clusters equipped with homogeneous high-speed Remote Direct Memory Access (RDMA) network interface cards (NICs). The acquisition and maintenance of such dedicated clusters is challenging. Current LLM training frameworks, like Megatron-LM and Megatron-DeepSpeed, focus primarily on optimizing training within homogeneous cluster settings. In this paper, we introduce Holmes, a training framework for LLMs that employs thoughtfully crafted data and model parallelism strategies over the heterogeneous NIC environment. Our primary technical contribution lies in a novel scheduling method that intelligently allocates distinct computational tasklets in LLM training to specific groups of GPU devices based on the characteristics of their connected NICs. Furthermore, our proposed framework, utilizing pipeline parallel techniques, demonstrates scalability to multiple GPU clusters, even in scenarios without high-speed interconnects between nodes in distinct clusters. We conducted comprehensive experiments that involved various scenarios in the heterogeneous NIC environment. In most cases, our framework achieves performance levels close to those achievable with homogeneous RDMA-capable networks (InfiniBand or RoCE), significantly exceeding training efficiency within the pure Ethernet environment. Additionally, we verified that our framework outperforms other mainstream LLM frameworks under heterogeneous NIC environment in terms of training efficiency and can be seamlessly integrated with them.

Figures (7)

• Figure 1: An overview of the Holmes framework. Holmes provides training support for a diverse range of LLM types and seamlessly integrates with contemporary mainstream LLM training frameworks. This flexibility extends to the training of LLMs in the heterogeneous NIC environment, where Holmes consistently achieves performance levels that are close to those achievable in the homogeneous NIC settings.
• Figure 2: An illustration of the scheduling problem in heterogeneous NIC environments, with the objective of minimizing communication costs for distributed LLM training tasks.
• Figure 3: An illustration of parallelism mechanism in Holmes. In this case, a transformer-based model with 6 layers is trained across 2 clusters. Each cluster comprises 2 nodes, each equipped with 4 GPU devices. Communication between Node 1 and Node 2 utilizes InfiniBand, while Node 3 and Node 4 use RoCE. However, there is no high-speed interconnect between the two clusters, and communication between them relies solely on low-speed Ethernet. For parallelism settings, the degrees of data, tensor, and pipeline parallelisms are 2, 2, and 4, respectively. Pipeline parallelism is implemented between the two clusters using Ethernet. The model's layers are unevenly partitioned into 2 stages and further distributed to different GPU devices. Data parallelism is performed within each cluster using RDMA, and tensor parallelism is implemented within each node using NVLink.
• Figure 4: The time cost of grads-reduce-scatter operation with different parameter groups in both homogeneous and heterogeneous NIC environments.
• Figure 5: Performance of different parameter groups in training GPT model with various pipeline partition strategies.