LoCA: Location-Aware Cosine Adaptation for Parameter-Efficient Fine-Tuning

TL;DR

LoCA addresses the inefficiency of full fine-tuning by introducing a frequency-domain, location-aware PEFT method based on the inverse discrete cosine transform. By reparameterizing weight updates as $W' = W_0 + \alpha C^{T} \mathcal{S}(\mathbf{a},\mathbf{l},\mathbf{1}) D$, and using discrete cosine transforms to avoid discrete optimization over signs, LoCA achieves expressive updates with a small set of learnable components. The method introduces a finite-difference gradient estimator for component locations and an alternating optimization schedule, enabling efficient learning across NLP and vision tasks where LoCA often matches or surpasses state-of-the-art PEFT methods with far fewer parameters. Theoretical analysis shows that strategically chosen frequency components can outperform low-rank approaches, and empirical results demonstrate robust performance gains across GLUE, E2E NLG, instruction tuning, and image classification, highlighting LoCA’s practical impact for scalable fine-tuning of large models.

Abstract

Low-rank adaptation (LoRA) has become a prevalent method for adapting pre-trained large language models to downstream tasks. However, the simple low-rank decomposition form may constrain the hypothesis space. To address this limitation, we introduce Location-aware Cosine Adaptation (LoCA), a novel frequency-domain parameter-efficient fine-tuning method based on inverse Discrete Cosine Transform (iDCT) with selective locations of learnable components. We begin with a comprehensive theoretical comparison between frequency-domain and low-rank decompositions for fine-tuning pre-trained large models. Our analysis reveals that frequency-domain decomposition with carefully selected frequency components can surpass the expressivity of traditional low-rank-based methods. Furthermore, we demonstrate that iDCT offers a more efficient implementation compared to inverse Discrete Fourier Transform (iDFT), allowing for better selection and tuning of frequency components while maintaining equivalent expressivity to the optimal iDFT-based adaptation. By employing finite-difference approximation to estimate gradients for discrete locations of learnable coefficients on the DCT spectrum, LoCA dynamically selects the most informative frequency components during training. Experiments on diverse language and vision fine-tuning tasks demonstrate that LoCA offers enhanced parameter efficiency while maintains computational feasibility comparable to low-rank-based methods.

Figures (15)

• Figure 1: Analysis of the weight incremental matrices. (a) Empirical distribution of the incremental query ($\Delta W_q$) and value ($\Delta W_v$) projection matrices for a representative middle layer. (b) p-values of the hypothesis test for $\Delta W_q$ and $\Delta W_v$ across different layers. (c) Empirical spectral density (ESD) of $\Delta W_q$ and $\Delta W_v$ for layer 4. Same phenomena are observed in other weight matrices.
• Figure 2: Evaluation loss (left) and performance (right) of our method with RoBERTa-base and ViT-base models. We record every 10 steps. The solid lines represent alternating optimization of coefficients and locations, while the dashed lines represent optimizing coefficients only.
• Figure 3: Performance comparison under different parameter budgets on QQP (RoBERTa-base) and FGVC (ViT-base).
• Figure 4: Influence of $\alpha$ and $\mathcal{B}_s$ on MRPC (RoBERTa-base).
• Figure 5: Empirical spectral density of the fine-tuned $W^{\prime}$ across multiple layers. The experimental settings are the same as those in Section \ref{['sec:Preliminary_Analysis']}.

Theorems & Definitions (17)

• Proposition 1
• Theorem 1
• Theorem 2
• Lemma 1
• proof
• Proposition 2
• proof
• Lemma 2: Discrete Parseval Theorem
• Lemma 3
• Lemma 4