Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Efficient Orthogonal Fine-Tuning with Principal Subspace Adaptation

Fei Wu, Jia Hu, Geyong Min, Shiqiang Wang

TL;DR

Efficient Orthogonal Fine-Tuning with Principal Subspace adaptation (PSOFT), which confines orthogonal transformations to the principal subspace of pre-trained weights and introduces efficient tunable vectors that gradually relax orthogonality during training to enhance adaptability.

Abstract

Driven by the rapid growth of model parameters, parameter-efficient fine-tuning (PEFT) has become essential for adapting large models to diverse downstream tasks under constrained computational resources. Within this paradigm, orthogonal fine-tuning and its variants preserve semantic representations of pre-trained models, but struggle to achieve both expressiveness and efficiency in terms of parameter counts, memory, and computation. To overcome this limitation, we propose efficient Orthogonal Fine-Tuning with Principal Subspace adaptation (PSOFT), which confines orthogonal transformations to the principal subspace of pre-trained weights. Specifically, PSOFT constructs this subspace via matrix decomposition to enable compatible transformations with higher effective rank, establishes a theoretical condition that strictly maintains the geometry of this subspace for essential semantic preservation, and introduces efficient tunable vectors that gradually relax orthogonality during training to enhance adaptability. Extensive experiments on 35 NLP and CV tasks across four representative models demonstrate that PSOFT offers a practical and scalable solution to simultaneously achieve semantic preservation, expressiveness, and multi-dimensional efficiency in PEFT. The code is publicly available at https://github.com/fei407/PSOFT.

Efficient Orthogonal Fine-Tuning with Principal Subspace Adaptation

TL;DR

Efficient Orthogonal Fine-Tuning with Principal Subspace adaptation (PSOFT), which confines orthogonal transformations to the principal subspace of pre-trained weights and introduces efficient tunable vectors that gradually relax orthogonality during training to enhance adaptability.

Abstract

Driven by the rapid growth of model parameters, parameter-efficient fine-tuning (PEFT) has become essential for adapting large models to diverse downstream tasks under constrained computational resources. Within this paradigm, orthogonal fine-tuning and its variants preserve semantic representations of pre-trained models, but struggle to achieve both expressiveness and efficiency in terms of parameter counts, memory, and computation. To overcome this limitation, we propose efficient Orthogonal Fine-Tuning with Principal Subspace adaptation (PSOFT), which confines orthogonal transformations to the principal subspace of pre-trained weights. Specifically, PSOFT constructs this subspace via matrix decomposition to enable compatible transformations with higher effective rank, establishes a theoretical condition that strictly maintains the geometry of this subspace for essential semantic preservation, and introduces efficient tunable vectors that gradually relax orthogonality during training to enhance adaptability. Extensive experiments on 35 NLP and CV tasks across four representative models demonstrate that PSOFT offers a practical and scalable solution to simultaneously achieve semantic preservation, expressiveness, and multi-dimensional efficiency in PEFT. The code is publicly available at https://github.com/fei407/PSOFT.
Paper Structure (36 sections, 3 theorems, 19 equations, 12 figures, 22 tables, 1 algorithm)

This paper contains 36 sections, 3 theorems, 19 equations, 12 figures, 22 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Preliminaries
  4. Methodology
  5. Dimension-Compatible Orthogonal Transforms
  6. Guaranteed Geometry Preservation in the Principal Subspace
  7. Efficient Relaxations of Orthogonality
  8. Experiments
  9. Discussion on Scaling to Larger Models
  10. Conclusion
  11. Algorithm of the proposed PSOFT
  12. Proof for the Angle and Norm Preservation Theorem
  13. Cayley Parameterization
  14. Comparison of Trainable Parameters for PEFT Methods
  15. The Activation Memory Statistics across Different PEFT Methods
  16. ...and 21 more sections

Key Result

Theorem 4.1

Let ${\bm{W}}_{\textnormal{pri}} = {\bm{A}} {\bm{B}}$ denote the principal weights and ${\bm{W}}_{\textnormal{ps-tuned}} = {\bm{A}} {\bm{R}} {\bm{B}}$ denote the fine-tuned weights. For ${\bm{W}}_{\textnormal{ps-tuned}}$ to preserve (i) pairwise angles between columns, and (ii) column norms of ${\bm

Figures (12)

  • Figure 1: Overview of the architectures of LoRA, OFT, and the proposed PSOFT.
  • Figure 2: Our proposed method: PSOFT. The left panel illustrates the principles of OFT variants. On the right, PSOFT preserves the angles and norms of ${\bm{W}}_{\text{pri}}$ (blue) in the fine-tuned ${\bm{W}}_{\text{ps-tuned}}$ (orange), while allowing adjustable angles and scalable norms in the sector.
  • Figure 3: Effect of tunable vectors.
  • Figure 4: (a) Memory usage across batch sizes. (b) Training speed across different models.
  • Figure 5: The architecture of a single transformer layer, including the attention layer and the feed forward network layer and self attention layer.
  • ...and 7 more figures

Theorems & Definitions (5)

  • Theorem 4.1: Informal: Angle and norm preservation in the principal subspace
  • Theorem B.1: Formal: Column-wise angle and norm preservation in the low-rank subspace
  • proof
  • Theorem C.1
  • proof