Towards Symmetric Low-Rank Adapters
Tales Panoutsos, Rodrygo L. T. Santos, Flavio Figueiredo
TL;DR
SymLoRA tackles the challenge of parameter-efficient fine-tuning for large pre-trained models by introducing Symmetric Low-Rank Adapters. It represents downstream-tuning updates with a spectral decomposition $Q \, diag(\Lambda)\, Q^T$, where $Q \in \mathbb{R}^{n\times r}$ and $\Lambda \in \mathbb{R}^r$, reducing the number of trainable weights by about half relative to standard LoRA's $BA$ formulation. The approach maintains downstream performance with negligible accuracy loss while improving memory and compute efficiency during training. This holds promise for deploying adaptable, resource-efficient models in constrained environments while preserving high task performance.
Abstract
In this paper, we introduce Symmetric Low-Rank Adapters, an optimized variant of LoRA with even fewer weights. This method utilizes Low-Rank Symmetric Weight Matrices to learn downstream tasks more efficiently. Traditional LoRA accumulates fine-tuning weights with the original pre-trained weights via a Singular Value Decomposition (SVD) like approach, i.e., model weights are fine-tuned via updates of the form $BA$ (where $B \in \mathbb{R}^{n\times r}$, $A \in \mathbb{R}^{r\times n}$, and $r$ is the rank of the merged weight matrix). In contrast, our approach, named SymLoRA, represents fine-tuning weights as a Spectral Decomposition, i.e., $Q \, diag(Λ)\, Q^T$, where $Q \in \mathbb{R}^{n\times r}$ and $Λ\in \mathbb{R}^r$. SymLoRA requires approximately half of the finetuning weights. Here, we show that this approach has negligible losses in downstream efficacy.