Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

ISO: Overlap of Computation and Communication within Seqenence For LLM Inference

Bin Xiao, Lei Su

TL;DR

This work tackles the inefficiency caused by the serial overlap of computation and communication in multi-GPU LLM inference under tensor parallelism. It introduces Intra-Sequence Overlap (ISO), a sequence-level strategy that splits a single request into two chunks to overlap attention-focused prefill computation with communication, thereby improving resource utilization. Empirical results on 30B and 70B models across 4090 and A800 GPUs show ISO reduces prefill time by about 35% on 4090 and 15% on A800, outperforming prior overlap methods. The findings highlight the importance of balancing computation and communication and point to adaptive segmentation as a path to further gains, while noting limited benefits during the decode stage and varying impact across hardware.

Abstract

In the realm of Large Language Model (LLM) inference, the inherent structure of transformer models coupled with the multi-GPU tensor parallelism strategy leads to a sequential execution of computation and communication. This results in substantial underutilization of computing resources during the communication phase. To mitigate this inefficiency, various techniques have been developed to optimize the use of computational power throughout the communication process. These strategies primarily involve overlapping matrix computations and communications, as well as interleaving micro-batches across different requests. Nonetheless, these approaches either fall short of achieving ideal overlap or impose certain limitations on their application. To overcome these challenges, this paper introduces a novel strategy for computation-communication overlap that operates at the sequence level. This method not only enhances the degree of overlap but also minimizes the constraints on its applicability. Experimental evaluations conducted using 30b/70b models have demonstrated significant improvements in efficiency. Specifically, the proposed technique has been shown to reduce time consumption by approximately 35% on 4090 GPU and by roughly 15% on A800 GPU during the prefill stage of LLM inference.

ISO: Overlap of Computation and Communication within Seqenence For LLM Inference

TL;DR

This work tackles the inefficiency caused by the serial overlap of computation and communication in multi-GPU LLM inference under tensor parallelism. It introduces Intra-Sequence Overlap (ISO), a sequence-level strategy that splits a single request into two chunks to overlap attention-focused prefill computation with communication, thereby improving resource utilization. Empirical results on 30B and 70B models across 4090 and A800 GPUs show ISO reduces prefill time by about 35% on 4090 and 15% on A800, outperforming prior overlap methods. The findings highlight the importance of balancing computation and communication and point to adaptive segmentation as a path to further gains, while noting limited benefits during the decode stage and varying impact across hardware.

Abstract

In the realm of Large Language Model (LLM) inference, the inherent structure of transformer models coupled with the multi-GPU tensor parallelism strategy leads to a sequential execution of computation and communication. This results in substantial underutilization of computing resources during the communication phase. To mitigate this inefficiency, various techniques have been developed to optimize the use of computational power throughout the communication process. These strategies primarily involve overlapping matrix computations and communications, as well as interleaving micro-batches across different requests. Nonetheless, these approaches either fall short of achieving ideal overlap or impose certain limitations on their application. To overcome these challenges, this paper introduces a novel strategy for computation-communication overlap that operates at the sequence level. This method not only enhances the degree of overlap but also minimizes the constraints on its applicability. Experimental evaluations conducted using 30b/70b models have demonstrated significant improvements in efficiency. Specifically, the proposed technique has been shown to reduce time consumption by approximately 35% on 4090 GPU and by roughly 15% on A800 GPU during the prefill stage of LLM inference.
Paper Structure (14 sections, 3 figures, 1 table)

This paper contains 14 sections, 3 figures, 1 table.

Table of Contents

  1. Introduction
  2. Background
  3. LLM Inference
  4. Overlap of Computation and Communication
  5. ISO
  6. Overlap strategy
  7. Asymmetric optimization strategy
  8. Evaluation
  9. Experiment Settings
  10. GPU Types, Models, and Baselines
  11. Metrics
  12. End-to-end Results
  13. Conclusion
  14. Discussion

Figures (3)

  • Figure 1: Overview of (a) Orginal pipeline, (b) Gemm overlp, (c) Request overlap, (d) ISO overlap.
  • Figure 2: Optimization Strategy of (a) Communication dominates, (b) Computation dominates.
  • Figure 3: Adaptive Attention and MLP Imbalance Splitting Strategy.