ARA: Adaptive Rank Allocation for Efficient Large Language Model SVD Compression

TL;DR

The paper tackles the challenge of allocating SVD ranks across linear modules of large language models under a global compression budget. It introduces Adaptive Rank Allocation (ARA), which uses a monotone probabilistic mask and a global-optimum guidance loss to jointly optimize per-module ranks and compression decisions, including a mechanism to preserve dense matrices when the compression ratio nears 1. ARA объединяет mask-generation, a full-rank guidance term, and a compression-rate loss into a single objective, trained with STE to align training and deployment behavior. Extensive experiments on LLaMA2-7B/13B and Qwen3-8B/14B show that ARA consistently outperforms state-of-the-art baselines in perplexity and zero-shot tasks at 80% and 60% compression, and remains robust across model sizes. The approach also demonstrates compatibility with quantization and fast inference, offering a practical route to highly compressed yet performant LLMs.

Abstract

In the field of large language model (LLM) compression, singular value decomposition (SVD) is a widely studied and adopted low-rank decomposition technique. Since SVD operates exclusively on linear modules, and these modules in LLMs are separated by nonlinear components, SVD can only be applied independently to each linear module. Under a global compression ratio constraint, determining the appropriate rank for different linear modules becomes a critical problem. Existing approaches, such as heuristic algorithms and mask-based training, have made progress in addressing this challenge. However, these methods still suffer from several limitations: heuristic algorithms explore the solution space within restricted regions, while mask-based training struggles to efficiently capture the relationship between singular value spectra and trainable parameters. More importantly, current methods overlook the key property that the gain function is non-smooth at a compression ratio of 1, which often leads the training process to suboptimal local minima. To address these issues, we propose an Adaptive Rank Allocation (ARA) method. Specifically, (1) ARA introduces a dedicated mask design that enables efficient mapping and updating between retained ranks and trainable parameters; and (2) it employs an additional loss function to guide parameter selection toward globally optimal solutions. Experimental results demonstrate that ARA achieves state-of-the-art performance. On the LLaMA2-7B model with a 80\% compression ratio, ARA reduces perplexity on WikiText2 from 8.38 to 6.42 and improves average zero-shot task accuracy by 9.72 percentage points compared with uniform compression. These results highlight the effectiveness of our method for rank allocation in SVD-based LLM compression.

Figures (7)

• Figure 1: A comparison of our masking strategy against prior work. Prior methods for mask design include (a) heuristic approaches with a limited set of patterns, (b) Gumbel-Sigmoid masks with independent parameters for each value, and (c) tanh-based masks controlled by a single soft cutoff parameter. In contrast, (d) our method utilizes a probabilistic mask to derive a final binary mask.
• Figure 2: A comparison of our training scope against prior work. (a) Whereas prior work can only find local optima, our method achieves the global optimum. (b) Previous methods operate within fixed low-rank or full-rank training scopes. (c) Our approach dynamically adapts the entire computational flow based on the target compression rate.
• Figure 3: An illustration of our ARA framework. Trainable parameters {$\alpha_i$} produce a probabilistic mask via a mapping matrix, from which a compression rate R and a binary mask M are derived. The rate R dynamically dictates the computational flow and activates a full-rank guidance loss when necessary. The entire model is trained with a joint objective comprising the model loss, the guidance loss, and a compression rate loss.
• Figure 4:
• Figure 5: Throughput (tokens/sec) for LLaMA2-7B with ARA and uniform at different compression rates, under (a) varying batch sizes and (b) varying sequence lengths.