High-Rank Structured Modulation for Parameter-Efficient Fine-Tuning

TL;DR

This work introduces SMoA, a high-rank parameter-efficient fine-tuning method that freezes the pretrained weights and modulates them across multiple disjoint singular subspaces. By partitioning the principal singular directions via cumulative spectral energy and applying fixed spectral modulation to each subspace, SMoA achieves a higher potential rank than LoRA without increasing parameter count. Theoretical rank analysis shows SMoA can reach up to $r d$ rank with $2 d r / K$ parameters, outperforming LoRA's bound under the same budget, while empirical results across commonsense reasoning, dialogue, and mathematical reasoning demonstrate state-of-the-art or near-state-of-the-art performance on Llama backbones. The approach offers a practical, scalable pathway for capacity-enhanced PEFT with tunable equivalent rank through the subspace count $K$, enabling effective adaptation under resource constraints.

Abstract

As the number of model parameters increases, parameter-efficient fine-tuning (PEFT) has become the go-to choice for tailoring pre-trained large language models. Low-rank Adaptation (LoRA) uses a low-rank update method to simulate full parameter fine-tuning, which is widely used to reduce resource requirements. However, decreasing the rank encounters challenges with limited representational capacity when compared to full parameter fine-tuning. We present \textbf{SMoA}, a high-rank \textbf{S}tructured \textbf{MO}dulation \textbf{A}dapter that uses fewer trainable parameters while maintaining a higher rank, thereby improving the model's representational capacity and offering improved performance potential. The core idea is to freeze the original pretrained weights and selectively amplify or suppress important features of the original weights across multiple subspaces. The subspace mechanism provides an efficient way to increase the capacity and complexity of a model. We conduct both theoretical analyses and empirical studies on various tasks. Experiment results show that SMoA outperforms LoRA and its variants on 10 tasks, with extensive ablation studies validating its effectiveness.

Figures (3)

• Figure 1: Comparison between LoRA (left) and and the proposed SMoA (right). The core idea of SMoA lies in diversifying the modulation of the original weights across multiple subspaces.
• Figure 2: Comparison of the rank of incremental weight $\Delta W$ for different PEFT methods. For MeLoRA and SMoA, the number of LoRA modules is set to 2.
• Figure 3: Performance of SMoA across tasks when $r$ increases ($K=2$).