Q3R: Quadratic Reweighted Rank Regularizer for Effective Low-Rank Training

TL;DR

Q3R introduces Quadratic Reweighted Rank Regularization, an IRLS-inspired, optimizer-friendly regularizer that majorizes a smoothed rank surrogate $F_{\\epsilon}(\\mathbf{W})$ to induce low-rank weights during pre-training and fine-tuning of Transformer models. By defining a reweighting operator $\\mathcal{R}_{\\mathbf{W}',\\epsilon}$ and a quadratic model $Q_{\\epsilon}$, the method provides a tractable gradient and a principled path toward controlled rank reduction, implemented via the AdamQ3R optimizer with periodic reweighting updates. Empirically, Q3R achieves substantial parameter reductions (e.g., up to $60\%$ in ViT-Tiny CIFAR-10) with minimal accuracy loss, and demonstrates competitive performance relative to LoRA/LoRITa in low-rank pre-training and GLUE-based fine-tuning across vision and language tasks. The work shows broad applicability to pre-training and fine-tuning, offering a unified, efficient approach to low-rank compression, while acknowledging limitations such as hyperparameter sensitivity and overhead that warrant future improvements.

Abstract

Parameter-efficient training, based on low-rank optimization, has become a highly successful tool for fine-tuning large deep-learning models. However, these methods fail at low-rank pre-training tasks where maintaining the low-rank structure and the objective remains a challenging task. We propose the Quadratic Reweighted Rank Regularizer dubbed Q3R, which leads to a novel low-rank inducing training strategy inspired by the iteratively reweighted least squares (IRLS) framework. Q3R is based on a quadratic regularizer term which majorizes a smoothed log determinant serving as rank surrogate objective. Unlike other low-rank training techniques, Q3R is able to train weight matrices with prescribed, low target ranks of models that achieve comparable predictive performance as dense models, with small computational overhead, while remaining fully compatible with existing architectures. For example, we demonstrated one experiment where we are able to truncate $60\%$ and $80\%$ of the parameters of a ViT-Tiny model with $~1.3\%$ and $~4\%$ accuracy drop in CIFAR-10 performance respectively. The efficacy of Q3R is confirmed on Transformers across both image and language tasks, including for low-rank fine-tuning.

Figures (4)

• Figure 1: Performance curves on CIFAR-10 with rank regularization applied to MLP and QKV blocks: (a) Best performance across methods, (b) AdamQ3R vs. LoRITa variants.
• Figure 2: Performance curves on CIFAR-100 with rank regularization applied to QKV blocks: (a) Best performance across methods, (b) AdamQ3R vs. LoRITa variants.
• Figure 3: Performance curves on CIFAR-10 with rank regularization applied to QKV blocks: (a) Best performance across methods, (b) AdamQ3R vs. LoRITa variants.
• Figure : Low-Rank Training via $\text{Adam}\texttt{Q3R}$

Theorems & Definitions (12)

• Definition 4.1: Reweighting Operator KuemmerleMaly-2023Recovering
• Lemma 4.1
• Definition A.1: Absolutely permutation symmetric functions
• Proposition A.1: Differentiability of Spectral Functions Lewis05_Nonsm1
• Lemma A.2
• proof : Proof of \ref{['lemma:gradient:Feps']}
• Remark A.3
• proof : Proof of \ref{['lemma:RO:properties']}.1
• proof : Proof of \ref{['lemma:RO:properties']}.2
• Lemma A.4: Gradient Condition