Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Spectral Adapter: Fine-Tuning in Spectral Space

Fangzhao Zhang, Mert Pilanci

TL;DR

This work studies the enhancement of current PEFT methods by incorporating the spectral information of pretrained weight matrices into the fine-tuning procedure, and shows that the proposed fine-tuning model enables better parameter efficiency and tuning performance as well as benefits multi-adapter fusion.

Abstract

Recent developments in Parameter-Efficient Fine-Tuning (PEFT) methods for pretrained deep neural networks have captured widespread interest. In this work, we study the enhancement of current PEFT methods by incorporating the spectral information of pretrained weight matrices into the fine-tuning procedure. We investigate two spectral adaptation mechanisms, namely additive tuning and orthogonal rotation of the top singular vectors, both are done via first carrying out Singular Value Decomposition (SVD) of pretrained weights and then fine-tuning the top spectral space. We provide a theoretical analysis of spectral fine-tuning and show that our approach improves the rank capacity of low-rank adapters given a fixed trainable parameter budget. We show through extensive experiments that the proposed fine-tuning model enables better parameter efficiency and tuning performance as well as benefits multi-adapter fusion.

Spectral Adapter: Fine-Tuning in Spectral Space

TL;DR

This work studies the enhancement of current PEFT methods by incorporating the spectral information of pretrained weight matrices into the fine-tuning procedure, and shows that the proposed fine-tuning model enables better parameter efficiency and tuning performance as well as benefits multi-adapter fusion.

Abstract

Recent developments in Parameter-Efficient Fine-Tuning (PEFT) methods for pretrained deep neural networks have captured widespread interest. In this work, we study the enhancement of current PEFT methods by incorporating the spectral information of pretrained weight matrices into the fine-tuning procedure. We investigate two spectral adaptation mechanisms, namely additive tuning and orthogonal rotation of the top singular vectors, both are done via first carrying out Singular Value Decomposition (SVD) of pretrained weights and then fine-tuning the top spectral space. We provide a theoretical analysis of spectral fine-tuning and show that our approach improves the rank capacity of low-rank adapters given a fixed trainable parameter budget. We show through extensive experiments that the proposed fine-tuning model enables better parameter efficiency and tuning performance as well as benefits multi-adapter fusion.
Paper Structure (27 sections, 2 theorems, 8 equations, 16 figures, 8 tables)

This paper contains 27 sections, 2 theorems, 8 equations, 16 figures, 8 tables.

Table of Contents

  1. Introduction
  2. Spectral Adapter: Incorporating Spectral Information into Fine-Tuning
  3. Theoretical Insights
  4. Adapter Rank Capacity
  5. Weight Subspace Alignment
  6. Empirical Results: The Impact of Spectral Information
  7. Language Model Fine-Tuning: Enhancing Fine-Tuning Results with Spectral AdapterA
  8. Diffusion Model Fusion: Improving Multi-Object Fine-Tuning with Spectral AdapterA
  9. Diffusion Model Expressiveness: Improving Parameter Efficiency with Spectral AdapterR
  10. Final Note: A Closer Look at SVD Cost
  11. Conclusion and Limitations
  12. Acknowledgement
  13. Prior Work
  14. Rank Capacity Proof
  15. Cayley Parameterization Proof
  16. ...and 12 more sections

Key Result

Lemma 3.1

Suppose that $W\in\mathbb{R}^{n\times m}$ is an arbitrary full row-rank matrix and $n\le m$ without loss of generality. Consider rank-r LoRA and rank-r additive spectral adapter, which have an equal number of trainable parameters. We have

Figures (16)

  • Figure 1: Training loss of fine-tuning Llama3 8B model with Orca Math dataset mitra2024orcamath and evaluation score on GSM8K benchmark cobbe2021training. We follow experimental setup in qdora, see Appendix \ref{['loss_image_detail']} for details. All methods except full fine-tuning maintain approximately $0.23\%$ trainable parameters.
  • Figure 2: Compared to LoRA which proposes to add low-rank trainable matrices to pretrained weights, we study two types of spectral adapters: Spectral Adapter$^A$ considers additively tuning the top columns of singular vector matrices and Spectral Adapter$^R$ considers orthogonally rotating the top columns of singular vector matrices.
  • Figure 3: Top singular vector of pretrained weight recognizes more ideal neuron direction. Illustration plot for Section \ref{['subspace_sec']}.
  • Figure 4: Distributing different concept tunings along different spectral space helps with identity preservation in multi-adapter fusion, see Section \ref{['a_exp']} for details.
  • Figure 5: Generation results of Chilloutmix diffusion model chillout with different fused adapters tuned on three custom animal concepts. See Section \ref{['fourobj']} for details.
  • ...and 11 more figures

Theorems & Definitions (4)

  • Lemma 3.1
  • Lemma 4.1
  • proof
  • proof