Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Parameter Efficient Quasi-Orthogonal Fine-Tuning via Givens Rotation

Xinyu Ma, Xu Chu, Zhibang Yang, Yang Lin, Xin Gao, Junfeng Zhao

TL;DR

This work targets parameter-efficient fine-tuning for large pretrained models by addressing OFT's two limitations: high parameter cost ($O(d^2)$) and restricted adaptation. It introduces GOFT, which uses a product of $d-1$ Givens rotations to achieve equivalent expressiveness to OFT with $O(d)$ parameters, augmented by a parallel rotation scheme that reduces computations to $O(\log d)$. Building on GOFT, qGOFT adds per-rotation norm and relative-angle flexibility under soft orthogonality regularization to better capture downstream semantic shifts. Empirical results across NLP and vision benchmarks (e.g., GLUE, SQuAD, VTAB) demonstrate strong performance at low parameter budgets, with negligible inference overhead since the learned rotations can be merged into the original weights at test time, highlighting the practical impact for scalable deployment of PEFT.

Abstract

With the increasingly powerful performances and enormous scales of pretrained models, promoting parameter efficiency in fine-tuning has become a crucial need for effective and efficient adaptation to various downstream tasks. One representative line of fine-tuning methods is Orthogonal Fine-tuning (OFT), which rigorously preserves the angular distances within the parameter space to preserve the pretrained knowledge. Despite the empirical effectiveness, OFT still suffers low parameter efficiency at $\mathcal{O}(d^2)$ and limited capability of downstream adaptation. Inspired by Givens rotation, in this paper, we proposed quasi-Givens Orthogonal Fine-Tuning (qGOFT) to address the problems. We first use $\mathcal{O}(d)$ Givens rotations to accomplish arbitrary orthogonal transformation in $SO(d)$ with provable equivalence, reducing parameter complexity from $\mathcal{O}(d^2)$ to $\mathcal{O}(d)$. Then we introduce flexible norm and relative angular adjustments under soft orthogonality regularization to enhance the adaptation capability of downstream semantic deviations. Extensive experiments on various tasks and pretrained models validate the effectiveness of our methods.

Parameter Efficient Quasi-Orthogonal Fine-Tuning via Givens Rotation

TL;DR

This work targets parameter-efficient fine-tuning for large pretrained models by addressing OFT's two limitations: high parameter cost () and restricted adaptation. It introduces GOFT, which uses a product of Givens rotations to achieve equivalent expressiveness to OFT with parameters, augmented by a parallel rotation scheme that reduces computations to . Building on GOFT, qGOFT adds per-rotation norm and relative-angle flexibility under soft orthogonality regularization to better capture downstream semantic shifts. Empirical results across NLP and vision benchmarks (e.g., GLUE, SQuAD, VTAB) demonstrate strong performance at low parameter budgets, with negligible inference overhead since the learned rotations can be merged into the original weights at test time, highlighting the practical impact for scalable deployment of PEFT.

Abstract

With the increasingly powerful performances and enormous scales of pretrained models, promoting parameter efficiency in fine-tuning has become a crucial need for effective and efficient adaptation to various downstream tasks. One representative line of fine-tuning methods is Orthogonal Fine-tuning (OFT), which rigorously preserves the angular distances within the parameter space to preserve the pretrained knowledge. Despite the empirical effectiveness, OFT still suffers low parameter efficiency at and limited capability of downstream adaptation. Inspired by Givens rotation, in this paper, we proposed quasi-Givens Orthogonal Fine-Tuning (qGOFT) to address the problems. We first use Givens rotations to accomplish arbitrary orthogonal transformation in with provable equivalence, reducing parameter complexity from to . Then we introduce flexible norm and relative angular adjustments under soft orthogonality regularization to enhance the adaptation capability of downstream semantic deviations. Extensive experiments on various tasks and pretrained models validate the effectiveness of our methods.
Paper Structure (39 sections, 1 theorem, 6 equations, 6 figures, 10 tables, 1 algorithm)

This paper contains 39 sections, 1 theorem, 6 equations, 6 figures, 10 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Works
  3. Preliminaries
  4. Orthogonal Fine-Tuning
  5. Givens Rotation
  6. Methodology
  7. Q1: Enhancing Parameter Efficiency of OFT in $SO(d)$ with Equivalent Expressive Power
  8. Q2: Enhancing Adaptation Capability
  9. Experiments
  10. Natural Language Understanding
  11. Instruction Tuning
  12. Experimental Setting
  13. Question Answering
  14. Experimental Setting
  15. Adapting Visual Foundation Models
  16. ...and 24 more sections

Key Result

Theorem 4.1

Given any vector $\bm{x} \in \mathbb{R}^d$, there always exist $d-1$ Givens rotations $\{\bm{G}(i_k, j_k; \theta_k)\}_{k=1}^{d-1}$ that can transform $\bm{x}$ to any vector $\bm{y} \in \mathbb{R}^d$ on the same sphere with $\bm{x}$, i.e., $\prod_{k=1}^{d-1} \bm{G}(i_k, j_k; \theta_k) \bm{x} = \bm{y}

Figures (6)

  • Figure 1: LoRA and OFT Reparameterization Tuning Methods.
  • Figure 2: Our proposed method: quasi-Givens Orthogonal Fine-Tuning (qGOFT). The left subfigure denotes the strict GOFT which applies $d-1$ Givens rotation to left-multiply with the pretrained weight matrix, where (a) and (b) depict the sequential and parallel rotation manner, respectively. The right subfigure illustrates how qGOFT works, where each Givens rotation in GOFT is substituted with a quasi-Givens matrix for norm and angular relaxation.
  • Figure 3: Win rate of GOFT versus other methods on GPT-4-turbo score of Vicuna-Eval benchmark.
  • Figure 4: Ablation Studies: (a) Comparisons of GOFT, qGOFT, and GOFT with only norm adjustment (GOFT*). (b) Varying orthogonal regularization strength $\lambda$ in qGOFT.
  • Figure 5: The illustrative example (i.e., rotating procedure) of our proof.
  • ...and 1 more figures

Theorems & Definitions (2)

  • Theorem 4.1
  • proof