InternEvo: Efficient Long-sequence Large Language Model Training via Hybrid Parallelism and Redundant Sharding

TL;DR

Long-sequence LLM training suffers from prohibitive activation memory and cross-node communication. InternEvo automatically searches a hierarchical space of memory management, hybrid parallelism, and independent sharding to minimize memory and communication costs, aided by an execution simulator and selective overlap. The system also mitigates GPU memory fragmentation and overlaps communication with computation, achieving up to 4.8x MFU gains over strong baselines on 7B–65B models with sequence lengths up to 256K. The results demonstrate scalable, efficient long-sequence training and practical applicability to state-of-the-art LLM workloads.

Abstract

Large language models (LLMs) with long sequences begin to power more and more fundamentally new applications we use every day. Existing methods for long-sequence LLM training are neither efficient nor compatible with commonly-used training algorithms such as FlashAttention. We design InternEvo to address these issues. InternEvo decouples all of the sharding dimensions into a new hierarchical space, and systematically analyzes the memory and communication cost of LLM training. Then, it generates an effective hybrid parallelism strategy. We design a new selective overlap mechanism to mitigate the communication overhead introduced by the hybrid parallelism. We also implement memory management techniques to reduce GPU memory fragmentation. Evaluation results show that InternEvo generates parallelization strategies that match or outperform existing methods in model FLOPs utilization.

Figures (15)

• Figure 1: Memory and communication costs of model states (parameters ($P$), gradients ($G$), and optimizer states ($OS$)) and activations (ACT) when training a 7B model with 32k sequence length on 128 GPUs. DS means DeepSpeed here.
• Figure 2:
• Figure 3: Search space overview. The orange and blue boxes are the dimensions in InternEvo's search space.
• Figure 4: Performance evaluation of collective communication operations using NCCL on Nvdia Ampere GPUs. For configurations with up to 8 GPUs, the tests were executed within a single node. For scenarios with more than 8 GPUs, we extended the evaluation across nodes interconnected by 4x Mellanox Infiniband HDR NICs.
• Figure 5: Three Sharding strategies.