Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Communication Compression for Tensor Parallel LLM Inference

Jan Hansen-Palmus, Michael Truong Le, Oliver Hausdörfer, Alok Verma

TL;DR

This paper tackles latency in tensor-parallel LLM inference by compressing inter-accelerator communications through fine-grained activation quantization. It introduces a block-wise, low-bit quantization scheme that compresses activation tensors after row-wise TP layers, balancing computational overhead with communication savings. Empirical results show about 3.3x compression with minimal perplexity increase and TTFT speedups up to 2x on slower interconnects, while highlighting the dependency on bandwidth and hardware. Building on microscaling and ocp_specification, the work provides a practical approach to accelerate inference in TP setups and outlines directions for future hardware-accelerated implementations and broader evaluation.

Abstract

Large Language Models (LLMs) have pushed the frontier of artificial intelligence but are comprised of hundreds of billions of parameters and operations. For faster inference latency, LLMs are deployed on multiple hardware accelerators through various Model Parallelism strategies. Our paper looks into the details on one such strategy - Tensor Parallel - and proposes to reduce latency by compressing inter-accelerator communication. We leverage fine grained quantization techniques to compress selected activations by 3.5 - 4.5x. Our proposed method leads up to 2x reduction of time-to-first-token (TTFT) with negligible model performance degradation.

Communication Compression for Tensor Parallel LLM Inference

TL;DR

This paper tackles latency in tensor-parallel LLM inference by compressing inter-accelerator communications through fine-grained activation quantization. It introduces a block-wise, low-bit quantization scheme that compresses activation tensors after row-wise TP layers, balancing computational overhead with communication savings. Empirical results show about 3.3x compression with minimal perplexity increase and TTFT speedups up to 2x on slower interconnects, while highlighting the dependency on bandwidth and hardware. Building on microscaling and ocp_specification, the work provides a practical approach to accelerate inference in TP setups and outlines directions for future hardware-accelerated implementations and broader evaluation.

Abstract

Large Language Models (LLMs) have pushed the frontier of artificial intelligence but are comprised of hundreds of billions of parameters and operations. For faster inference latency, LLMs are deployed on multiple hardware accelerators through various Model Parallelism strategies. Our paper looks into the details on one such strategy - Tensor Parallel - and proposes to reduce latency by compressing inter-accelerator communication. We leverage fine grained quantization techniques to compress selected activations by 3.5 - 4.5x. Our proposed method leads up to 2x reduction of time-to-first-token (TTFT) with negligible model performance degradation.

Paper Structure

This paper contains 18 sections, 1 figure, 5 tables.

Table of Contents

  1. Introduction
  2. Background
  3. Parallel Inference
  4. Model Quantization
  5. Related Work
  6. KV-cache Quantization
  7. Weight and Activation Quantization
  8. Method
  9. Communication Compression
  10. Model Evaluation
  11. Profiling
  12. Experiments
  13. Optimal Compression Scheme Search
  14. Measuring TTFT Speedups
  15. SoTA Comparison
  16. ...and 3 more sections

Figures (1)

  • Figure 1: An illustration of transformer-based LLM model parallelized using TP. In Figure \ref{['fig:model_flow']}, column-wise and row-wise TP layers are marked blue and red, respectively. Before reduction, we propose to compress the all_gather collective op (orange in Figure \ref{['fig:model_flow']}), as presented in Figure \ref{['fig:flow']}.