Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Differentially Private Subspace Fine-Tuning for Large Language Models

Lele Zheng, Xiang Wang, Tao Zhang, Yang Cao, Ke Cheng, Yulong Shen

TL;DR

The paper introduces DP-SFT, a two-stage method for differentially private fine-tuning of large language models that confines noise to a small, task-relevant gradient subspace. Stage 1 extracts a low-dimensional subspace from short, private gradient trajectories via SVD; Stage 2 performs gradient updates within this subspace, adding DP noise in $\mathbb{R}^k$ and mapping back to the full parameter space, achieving $(\varepsilon, \delta)$-DP with substantially reduced noise. Empirical results across SST-2, IMDB, QNLI, and MNLI show DP-SFT outperforms full-DP and LoRA/Adapter baselines under DP budgets, with strong stability and convergence benefits; subspaces learned from public data transfer across tasks with minimal loss. The approach delivers near-non-private utility under DP, offering a scalable and privacy-efficient pathway for adapting LLMs to downstream tasks in privacy-sensitive settings.

Abstract

Fine-tuning large language models on downstream tasks is crucial for realizing their cross-domain potential but often relies on sensitive data, raising privacy concerns. Differential privacy (DP) offers rigorous privacy guarantees and has been widely adopted in fine-tuning; however, naively injecting noise across the high-dimensional parameter space creates perturbations with large norms, degrading performance and destabilizing training. To address this issue, we propose DP-SFT, a two-stage subspace fine-tuning method that substantially reduces noise magnitude while preserving formal DP guarantees. Our intuition is that, during fine-tuning, significant parameter updates lie within a low-dimensional, task-specific subspace, while other directions change minimally. Hence, we only inject DP noise into this subspace to protect privacy without perturbing irrelevant parameters. In phase one, we identify the subspace by analyzing principal gradient directions to capture task-specific update signals. In phase two, we project full gradients onto this subspace, add DP noise, and map the perturbed gradients back to the original parameter space for model updates, markedly lowering noise impact. Experiments on multiple datasets demonstrate that DP-SFT enhances accuracy and stability under rigorous DP constraints, accelerates convergence, and achieves substantial gains over DP fine-tuning baselines.

Differentially Private Subspace Fine-Tuning for Large Language Models

TL;DR

The paper introduces DP-SFT, a two-stage method for differentially private fine-tuning of large language models that confines noise to a small, task-relevant gradient subspace. Stage 1 extracts a low-dimensional subspace from short, private gradient trajectories via SVD; Stage 2 performs gradient updates within this subspace, adding DP noise in and mapping back to the full parameter space, achieving -DP with substantially reduced noise. Empirical results across SST-2, IMDB, QNLI, and MNLI show DP-SFT outperforms full-DP and LoRA/Adapter baselines under DP budgets, with strong stability and convergence benefits; subspaces learned from public data transfer across tasks with minimal loss. The approach delivers near-non-private utility under DP, offering a scalable and privacy-efficient pathway for adapting LLMs to downstream tasks in privacy-sensitive settings.

Abstract

Fine-tuning large language models on downstream tasks is crucial for realizing their cross-domain potential but often relies on sensitive data, raising privacy concerns. Differential privacy (DP) offers rigorous privacy guarantees and has been widely adopted in fine-tuning; however, naively injecting noise across the high-dimensional parameter space creates perturbations with large norms, degrading performance and destabilizing training. To address this issue, we propose DP-SFT, a two-stage subspace fine-tuning method that substantially reduces noise magnitude while preserving formal DP guarantees. Our intuition is that, during fine-tuning, significant parameter updates lie within a low-dimensional, task-specific subspace, while other directions change minimally. Hence, we only inject DP noise into this subspace to protect privacy without perturbing irrelevant parameters. In phase one, we identify the subspace by analyzing principal gradient directions to capture task-specific update signals. In phase two, we project full gradients onto this subspace, add DP noise, and map the perturbed gradients back to the original parameter space for model updates, markedly lowering noise impact. Experiments on multiple datasets demonstrate that DP-SFT enhances accuracy and stability under rigorous DP constraints, accelerates convergence, and achieves substantial gains over DP fine-tuning baselines.
Paper Structure (23 sections, 4 theorems, 7 equations, 2 figures, 6 tables, 2 algorithms)

This paper contains 23 sections, 4 theorems, 7 equations, 2 figures, 6 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Related Works
  3. Preliminaries
  4. Differential Privacy
  5. Gaussian Mechanism in Deep Learning
  6. Methods
  7. Stage 1: Subspace Construction
  8. Stage 2: Private Subspace Training
  9. Privacy Guarantee
  10. Subspace Transferability
  11. Experiments
  12. Experimental Setup
  13. Datasets
  14. Base Model
  15. Baselines
  16. ...and 8 more sections

Key Result

Theorem 1

Let mechanisms $M_i$$( i \in \{1, 2, \ldots, T\} )$ satisfy $(\varepsilon_i, \delta_i)$-DP, then their combination $(M_1, M_2, \ldots, M_T)$ satisfies $(\sum_{i=1}^T \varepsilon_i, \sum_{i=1}^T \delta_i)$-DP.

Figures (2)

  • Figure 1: Fine-tuning Process of Large Language Models
  • Figure 2: An overview of the proposed DP-SFT. DP-SFT consists of two stages: (1) Subspace Construction: a set of model snapshots during private training is collected and stacked into a trajectory matrix, from which a task-specific subspace is extracted via SVD. (2) Private Subspace Training: gradients are computed and projected into the subspace, where calibrated DP noise is added before mapping them back to the full parameter space for model updates.

Theorems & Definitions (5)

  • Definition 1: $(\varepsilon, \delta)$-Differential Privacy
  • Theorem 1: Composition
  • Theorem 2: Post-processing invariance
  • Corollary 1: Property of Gaussian noise
  • Theorem 3: Differential Privacy of DP-SFT