D-com: Accelerating Iterative Processing to Enable Low-rank Decomposition of Activations

TL;DR

The paper introduces D-com, a co-designed approach to accelerate low-rank decomposition of activations in large language models by combining input activation decomposition with Lanczos-based SVD and a dedicated accelerator. It demonstrates that input decomposition can beat traditional expectations when paired with output-preserving strategies and channel-wise outlier extraction, achieving substantial end-to-end latency reductions (up to 22%) with modest model-quality degradation. The proposed D-com architecture enables compute expansion to mitigate memory bottlenecks, delivering 3.8× per-layer and up to 8.74× end-to-end speedups in simulations, along with favorable area and power characteristics compared to GPU-only implementations. Overall, the work shows that activating decomposition, when algorithmically and hardware co-optimized, can be a practical route to memory- and compute-efficient LLM inference at scale.

Abstract

The computation and memory costs of large language models kept increasing over last decade, which reached over the scale of 1T parameters. To address the challenges from the large scale models, model compression techniques such as low-rank decomposition have been explored. Previous model decomposition works have focused on weight decomposition to avoid costly runtime decomposition, whose latency often significantly exceeds the benefits from decomposition (e.g., 38% more end-to-end latency when running Llama2-7b on A100 with 4K sequence length with activation decomposition compared to no decomposition). In this work, we debunk such observations and report that the input decomposition can be significantly beneficial with a proper choice of decomposition algorithm and hardware support. We adopt progressive decomposition algorithm, Lanczos algorithm, and design a co-accelerator architecture for the decomposition algorithm. To address the memory- boundness of the decomposition operation, we introduce a novel compute replication methodology that moves the op- eration toward compute-bound region, which enables 6.2x speedup in our evaluation. We also develop an output shape- preserving computation scheme that eliminates decomposi- tion costs in consecutive layers. To compensate model quality loss from compression, we introduce a multi-track decom- position approach that separately handles outlier channels for high accuracy and low perplexity with minimal compu- tational costs. Combined together, our accelerator, D-com, provides 22% end-to-end latency improvements compared to A100 GPU at the cost of small model quality degradation (e.g., 3% on AI2 Reasoning Challenge task).

Figures (13)

• Figure 1: An overview of D-com.
• Figure 2: Decomposition algorithms convergence speed on comparison a single A100 GPU. The input matrix size is [4096, 468].
• Figure 3: Lanczos bidiagonal algorithm runtime breakdown on a single A100 GPU.
• Figure 4: Comparison of Llama2-7b layer inference runtime for different input sequence lengths. Batch size is 64 and K(number of Lanczos iterations is 10).
• Figure 5: Different decomposition strategies of a matmul layer in Large Language Models (LLMs) with output-decomposed computation.