Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

LEXI: Lossless Exponent Coding for Efficient Inter-Chiplet Communication in Hybrid LLMs

Miao Sun, Alish Kanani, Kaushik Shroff, Umit Ogras

Abstract

Data movement overheads increase the inference latency of state-of-the-art large language models (LLMs). These models commonly use the bfloat16 (BF16) format for stable training. Floating-point standards allocate eight bits to the exponent, but our profiling reveals that exponent streams exhibit fewer than 3 bits Shannon entropy, indicating high inherent compressibility. To exploit this potential, we propose LEXI, a novel lossless exponent compression scheme based on Huffman coding. LEXI compresses activations and caches on the fly while storing compressed weights for just-in-time decompression near compute, without sacrificing system throughput and model accuracy. The codecs at the ingress and egress ports of network-on-chip routers sustain the maximum link bandwidth via multi-lane LUT decoders, incurring only 0.09 percent area and energy overheads with GF 22 nm technology. LEXI reduces inter-chiplet communication and end-to-end inference latencies by 33-45 percent and 30-35 percent on modern Jamba, Zamba, and Qwen LLMs implemented on a homogeneous chiplet architecture.

LEXI: Lossless Exponent Coding for Efficient Inter-Chiplet Communication in Hybrid LLMs

Abstract

Data movement overheads increase the inference latency of state-of-the-art large language models (LLMs). These models commonly use the bfloat16 (BF16) format for stable training. Floating-point standards allocate eight bits to the exponent, but our profiling reveals that exponent streams exhibit fewer than 3 bits Shannon entropy, indicating high inherent compressibility. To exploit this potential, we propose LEXI, a novel lossless exponent compression scheme based on Huffman coding. LEXI compresses activations and caches on the fly while storing compressed weights for just-in-time decompression near compute, without sacrificing system throughput and model accuracy. The codecs at the ingress and egress ports of network-on-chip routers sustain the maximum link bandwidth via multi-lane LUT decoders, incurring only 0.09 percent area and energy overheads with GF 22 nm technology. LEXI reduces inter-chiplet communication and end-to-end inference latencies by 33-45 percent and 30-35 percent on modern Jamba, Zamba, and Qwen LLMs implemented on a homogeneous chiplet architecture.
Paper Structure (18 sections, 1 equation, 7 figures, 4 tables)

This paper contains 18 sections, 1 equation, 7 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Exponent Statistics Profiling and Motivation
  4. Profiling Methodology and Illustrative Results
  5. Motivation and Design Challenges
  6. LEXI: Entropy-Aware Exponent Compression
  7. Definitions and LEXI Architecture Overview
  8. Runtime Huffman Tree Generation
  9. Parallel Histogram Generation:
  10. Pipelined Codebook Generation Hardware
  11. Parallel Stream Encoding and Transmission
  12. On-the-fly Decompression Circuit
  13. Implementation and Evaluation of LEXI
  14. Experimental Setup
  15. Area-Performance Trade-off Optimization
  16. ...and 3 more sections

Figures (7)

  • Figure 1: Exponent compression opportunities on NVIDIA GeForce RTX 3090 NVIDIA_RTX3090. (a) BF16 exponents show only $\sim$3 bits of Shannon entropy and span fewer than 32 values, while mantissas consistently use the full 7-bit range. (b) Exponent compression shrinks weights from 422 MB to 151 MB and activations/caches from 360 MB to 155 MB. (c) Lossless exponent compression reduces communication overhead by 36-40% across Mamba, Transformer, and MoE blocks.
  • Figure 2: Overview for proposed compression/decompression units in chiplet system.
  • Figure 3: Hardware microarchitecture of LEXI: (a) compression circuit with $M$ parallel lanes, where per-lane local caches accelerate Huffman tree creation and a simple lookup table encodes exponents; (b) Four-stage decompression unit, each stage containing eight entries indexed by 8-, 16-, 24- and 32-bit indices.
  • Figure 4: Local cache hit rate vs. cache depth on WikiText-2 for three models: (a) Jamba, (b) Zamba, (c) Qwen
  • Figure 5: Codebook generation latency vs. cache size with 512 activations in BF16.
  • ...and 2 more figures