R-Sparse: Rank-Aware Activation Sparsity for Efficient LLM Inference

TL;DR

R-Sparse presents a training-free, rank-aware activation sparsity framework for LLM inference that exploits input-channel sparsity and offline low-rank weight decompositions to approximate full linear layer computations. By avoiding active-channel prediction and non-ReLU retraining, it achieves 50% model-level sparsity with accuracy near the full model across Llama-2/3 and Mistral, and delivers up to 43% end-to-end speedups with a customized kernel. The approach includes an evolutionary search to tailor sparse-rank recipes per layer and is compatible with quantization, enabling practical edge-device deployment. These results suggest that high sparsity in both attention and MLP blocks can be achieved without performance loss, significantly reducing compute and memory bottlenecks in on-device LLM inference.

Abstract

Large Language Models (LLMs), while demonstrating remarkable capabilities across various applications, present significant challenges during inference due to their substantial model size, especially when deployed on edge devices. Activation sparsity offers a promising solution to reduce computation and memory movement, enabling more efficient inference, particularly for small-batch on-device applications. However, current approaches face limitations with non-ReLU activation function, which are foundational to most advanced LLMs, or require heavy continual training. Additionally, the difficulty in predicting active channels and limited achievable sparsity ratios constrain the effectiveness of activation sparsity-based methods. In this paper, we introduce R-Sparse, a training-free activation sparsity approach capable of achieving high sparsity levels in advanced LLMs. We conducted two preliminary investigations into how different components contribute to the output within a single linear layer and found two key observations: (i) the non-sparse components of the input function can be regarded as a few bias terms, and (ii) The full computation can be effectively approximated by an appropriate combination of input channels and weight singular values. Building on this, we replace the linear layers in LLMs with a rank-aware sparse inference method that leverages the sparsity of input channels and singular value components, eliminating the need for active channel prediction like the output sparsity based approaches. Experiments on Llama-2/3 and Mistral models across ten diverse tasks demonstrate that R-Sparse achieves comparable performance at 50% model-level sparsity, resulting in a significant 43% end-to-end efficient improvements with customized kernels.

Figures (9)

• Figure 1: Contributions of each input channel and singular value components. The measurement metric is detailed in Section \ref{['singluar_eq']}. Results are obtained from Llama-2-7B with 16 training samples from C4. Both the input channel and SVD components are sorted from small to large for better visualization.
• Figure 2: Accuracy of Llama-2-7B on OpenBookQA OpenBookQA2018 (OBQA) and ARC Challenge allenai:arc (ARC-C) tasks.
• Figure 3: Importance of each input channel and singular value. Zoom in for better visualization. Results are obtained with the pretrained Llama-2-7B model and 16 samples from the C4 training dataset, each with a sequence length of 4096. Each subfigure corresponds to the results of different layers, with the horizontal axis representing the input channel index and the vertical axis representing the singular value index. The top, middle, and bottom subfigures represent the results of the first, middle, and last layers, respectively.
• Figure 4: Illustration of various compression techniques with corresponding impact on different input channels and singular values. The horizontal axis of the heatmap represents the input channels, while the vertical axis corresponds to the singular value index.
• Figure 5: Comparison results of Llama-2-7B across different model-level sparsity ratios on common-sense reasoning, language modeling and summarization tasks.