Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

FRoD: Full-Rank Efficient Fine-Tuning with Rotational Degrees for Fast Convergence

Guoan Wan, Tianyu Chen, Fangzheng Feng, Haoyi Zhou, Runhua Xu

TL;DR

FRoD tackles slow convergence and limited adaptation in parameter-efficient fine-tuning by marrying a hierarchical joint decomposition initialization with sparse rotational perturbations that enable full-rank updates at low parameter cost. The method extracts a global shared basis across layers and categories, then introduces sparse off-axis perturbations to expand the update space while preserving the dominant spectral directions. Empirically, FRoD matches or exceeds full fine-tuning accuracy on 20 vision, reasoning, and language benchmarks, converging in as few as 1–4 epochs with only 1.72% of trainable parameters, and exhibits robust performance across seeds. The work provides theoretical support via spectral stability and orthogonal decomposition of updates, along with practical insights into loss-landscape geometry and hyperparameter interactions, signaling meaningful gains for rapid, efficient adaptation of large models.

Abstract

Parameter-efficient fine-tuning (PEFT) methods have emerged as a practical solution for adapting large foundation models to downstream tasks, reducing computational and memory costs by updating only a small subset of parameters. Among them, approaches like LoRA aim to strike a balance between efficiency and expressiveness, but often suffer from slow convergence and limited adaptation capacity due to their inherent low-rank constraints. This trade-off hampers the ability of PEFT methods to capture complex patterns needed for diverse tasks. To address these challenges, we propose FRoD, a novel fine-tuning method that combines hierarchical joint decomposition with rotational degrees of freedom. By extracting a globally shared basis across layers and injecting sparse, learnable perturbations into scaling factors for flexible full-rank updates, FRoD enhances expressiveness and efficiency, leading to faster and more robust convergence. On 20 benchmarks spanning vision, reasoning, and language understanding, FRoD matches full model fine-tuning in accuracy, while using only 1.72% of trainable parameters under identical training budgets.

FRoD: Full-Rank Efficient Fine-Tuning with Rotational Degrees for Fast Convergence

TL;DR

FRoD tackles slow convergence and limited adaptation in parameter-efficient fine-tuning by marrying a hierarchical joint decomposition initialization with sparse rotational perturbations that enable full-rank updates at low parameter cost. The method extracts a global shared basis across layers and categories, then introduces sparse off-axis perturbations to expand the update space while preserving the dominant spectral directions. Empirically, FRoD matches or exceeds full fine-tuning accuracy on 20 vision, reasoning, and language benchmarks, converging in as few as 1–4 epochs with only 1.72% of trainable parameters, and exhibits robust performance across seeds. The work provides theoretical support via spectral stability and orthogonal decomposition of updates, along with practical insights into loss-landscape geometry and hyperparameter interactions, signaling meaningful gains for rapid, efficient adaptation of large models.

Abstract

Parameter-efficient fine-tuning (PEFT) methods have emerged as a practical solution for adapting large foundation models to downstream tasks, reducing computational and memory costs by updating only a small subset of parameters. Among them, approaches like LoRA aim to strike a balance between efficiency and expressiveness, but often suffer from slow convergence and limited adaptation capacity due to their inherent low-rank constraints. This trade-off hampers the ability of PEFT methods to capture complex patterns needed for diverse tasks. To address these challenges, we propose FRoD, a novel fine-tuning method that combines hierarchical joint decomposition with rotational degrees of freedom. By extracting a globally shared basis across layers and injecting sparse, learnable perturbations into scaling factors for flexible full-rank updates, FRoD enhances expressiveness and efficiency, leading to faster and more robust convergence. On 20 benchmarks spanning vision, reasoning, and language understanding, FRoD matches full model fine-tuning in accuracy, while using only 1.72% of trainable parameters under identical training budgets.
Paper Structure (18 sections, 8 theorems, 82 equations, 10 figures, 13 tables)

This paper contains 18 sections, 8 theorems, 82 equations, 10 figures, 13 tables.

Table of Contents

  1. Introduction
  2. Background and Motivation
  3. Background: LoRA and Its Variants
  4. Motivation: Convergence Efficiency
  5. Motivation: Parameter Degree of Freedom
  6. Method
  7. Hierarchical Joint Decomposition
  8. PEFT with Rotational Degree of Freedom
  9. Memory Usage of Projection Matrices
  10. Experiments
  11. Baselines
  12. Datasets
  13. Main Results
  14. Convergence Speed
  15. Analysis of Loss Landscape Geometry
  16. ...and 3 more sections

Key Result

Theorem 1

Let $\Sigma_i\in\mathbb{R}^{n\times n}$ be a diagonal matrix of singular values and let $S_i\in\mathbb{R}^{n\times n}$ be a perturbation such that Then, for every $k$, so the dominant singular directions encoded by $\Sigma_i$ are preserved up to a deviation bounded by $\|S_i\|_2$.

Figures (10)

  • Figure 1: Update Space and Rank: FRoD (joint matrix decomposition and off-axis sparse spaces) vs. SVD-based and random PEFT methods.
  • Figure 2: Comparison of loss landscapes for four representative fine-tuning methods in a principal parameter subspace. Each subplot shows the loss surface as a function of two principal directions ($\alpha$ and $\beta$) in parameter space, with the vertical axis denoting the loss value. The top row (a–d) presents landscapes at model initialization; the bottom row (e–h) shows them after convergence. These visualizations provide a comparative geometric view of how various fine-tuning strategies shape the optimization landscape during training.
  • Figure 3: Overview of FRoD. FRoD is a two-stage approach: hierarchical joint decomposition initializes the model by extracting a global shared basis, and sparse perturbations introduce rotational degrees of freedom during fine-tuning.
  • Figure 4: Training loss curves of Different LoRA methods and Full Fine-tuning on Cars. The shaded areas in the figure represent the error bounds of different methods. We randomly sample five seed values.
  • Figure 5: Visualization of Loss Landscapes for FRoD in Model Parameter Space.
  • ...and 5 more figures

Theorems & Definitions (10)

  • Theorem 1: Spectral Stability Under Sparse Perturbations
  • Corollary 2: Orthogonal Decomposition of Updates
  • Corollary 3: Angular Representation of the Total Update
  • Proposition 4: Non-Existence for a Joint Decomposition with a Common Orthogonal Matrix
  • proof
  • Lemma 5
  • Theorem 6: Spectral Stability Under Sparse Perturbations
  • Corollary 7: Orthogonal Decomposition of Updates
  • Corollary 8: Angular Representation of the Total Update
  • proof