FedSVD: Adaptive Orthogonalization for Private Federated Learning with LoRA
Seanie Lee, Sangwoo Park, Dong Bok Lee, Dominik Wagner, Haebin Seong, Tobias Bocklet, Juho Lee, Sung Ju Hwang
TL;DR
This paper tackles the challenge of privately fine-tuning large language models in federated settings using LoRA by addressing noise amplification under DP-SGD. It introduces FedSVD, a global reparameterization that periodically reinitializes the LoRA matrix A via the SVD of the aggregated BA, while allowing clients to update only B; this preserves privacy through post-processing and yields orthonormal A that tightens gradient bounds and improves conditioning. Theoretical analysis connects the SVD-based reparameterization to improved Hessian conditioning, and extensive experiments on GLUE tasks and HellaSwag demonstrate stronger accuracy and faster convergence than baselines in both private and non-private regimes, including compatibility with DoRA. Overall, FedSVD provides a simple, effective, and privacy-preserving mechanism for adaptive low-rank fine-tuning in federated learning, with practical gains in stability and performance. The approach balances privacy, communication efficiency, and learning capacity, making DP-enabled FL with LoRA more viable for real-world deployments.
Abstract
Low-Rank Adaptation (LoRA), which introduces a product of two trainable low-rank matrices into frozen pre-trained weights, is widely used for efficient fine-tuning of language models in federated learning (FL). However, when combined with differentially private stochastic gradient descent (DP-SGD), LoRA faces substantial noise amplification: DP-SGD perturbs per-sample gradients, and the matrix multiplication of the LoRA update ($BA$) intensifies this effect. Freezing one matrix (e.g., $A$) reduces the noise but restricts model expressiveness, often resulting in suboptimal adaptation. To address this, we propose $\texttt{FedSVD}$, a simple yet effective method that introduces a global reparameterization based on singular value decomposition (SVD). In our approach, each client optimizes only the $B$ matrix and transmits it to the server. The server aggregates the $B$ matrices, computes the product $BA$ using the previous $A$, and refactorizes the result via SVD. This yields a new adaptive $A$ composed of the orthonormal right singular vectors of $BA$, and an updated $B$ containing the remaining SVD components. This reparameterization avoids quadratic noise amplification, while allowing $A$ to better capture the principal directions of the aggregate updates. Moreover, the orthonormal structure of $A$ bounds the gradient norms of $B$ and preserves more signal under DP-SGD, as confirmed by our theoretical analysis. As a result, $\texttt{FedSVD}$ consistently improves stability and performance across a variety of privacy settings and benchmarks, outperforming relevant baselines under both private and non-private regimes.