LoRA and Privacy: When Random Projections Help (and When They Don't)
Yaxi Hu, Johanna Düngler, Bernhard Schölkopf, Amartya Sanyal
TL;DR
This work introduces the Wishart projection mechanism as a principled way to inject randomness into LoRA-style, parameter-efficient fine-tuning and analyzes its differential privacy properties. It shows non-asymptotic DP guarantees for vector-valued outputs without additive noise, but establishes a sharp negative result for matrix-valued queries in the noiseless setting, indicating LoRA alone is not private. A noisy variant reveals privacy amplification from the projection’s randomness and its low-rank structure, with stronger guarantees in both large-$r$ and small-$r$ regimes than additive noise alone. The results imply that LoRA updates are not private by default, but that carefully calibrated low-rank projections can improve privacy-utility trade-offs, enabling tighter privacy accounting and potentially lower noise for practical accuracy. Preliminary experiments corroborate the theoretical insights, showing scenario-dependent privacy amplification and utility gains when employing the noisy projection mechanism in DP-fine-tuning tasks.
Abstract
We introduce the (Wishart) projection mechanism, a randomized map of the form $S \mapsto M f(S)$ with $M \sim W_d(1/r I_d, r)$ and study its differential privacy properties. For vector-valued queries $f$, we prove non-asymptotic DP guarantees without any additive noise, showing that Wishart randomness alone can suffice. For matrix-valued queries, however, we establish a sharp negative result: in the noise-free setting, the mechanism is not DP, and we demonstrate its vulnerability by implementing a near perfect membership inference attack (AUC $> 0.99$). We then analyze a noisy variant and prove privacy amplification due to randomness and low rank projection, in both large- and small-rank regimes, yielding stronger privacy guarantees than additive noise alone. Finally, we show that LoRA-style updates are an instance of the matrix-valued mechanism, implying that LoRA is not inherently private despite its built-in randomness, but that low-rank fine-tuning can be more private than full fine-tuning at the same noise level. Preliminary experiments suggest that tighter accounting enables lower noise and improved accuracy in practice.