Privacy Enhanced PEFT: Tensor Train Decomposition Improves Privacy Utility Tradeoffs under DP-SGD

TL;DR

This work tackles the privacy risks of fine-tuning large language models on sensitive data by introducing TTLoRA, a tensor-train based adapter that dramatically reduces trainable parameters while preserving expressivity. It develops TTLoRA-DP, a differential privacy framework that extends ghost clipping to TT cores, enabling efficient DP-SGD for TTLoRA. Across Enron and PTB datasets, TTLoRA-DP consistently yields stronger privacy (lower membership leakage) than LoRA under identical DP budgets while maintaining comparable or better utility, and TTLoRA shows substantial inherent privacy advantages even without DP. The results demonstrate that TT decomposition not only compresses parameters but also acts as a principled architectural bias that improves the privacy–utility frontier for private LLM adaptation, offering a practical path toward privacy-preserving PEFT at ultra-low parameter counts.

Abstract

Fine-tuning large language models on sensitive data poses significant privacy risks, as membership inference attacks can reveal whether individual records were used during training. While Differential Privacy (DP) provides formal protection, applying DP to conventional Parameter-Efficient Fine-Tuning (PEFT) methods such as Low-Rank Adaptation (LoRA) often incurs substantial utility loss. In this work, we show that a more structurally constrained PEFT architecture, Tensor Train Low-Rank Adaptation (TTLoRA), can improve the privacy-utility tradeoff by shrinking the effective parameter space while preserving expressivity. To this end, we develop TTLoRA-DP, a differentially private training framework for TTLoRA. Specifically, we extend the ghost clipping algorithm to Tensor Train cores via cached contraction states, enabling efficient Differentially Private Stochastic Gradient Descent (DP-SGD) with exact per-example gradient norm computation without materializing full per-example gradients. Experiments on GPT-2 fine-tuning over the Enron and Penn Treebank datasets show that TTLoRA-DP consistently strengthens privacy protection relative to LoRA-DP while maintaining comparable or better downstream utility. Moreover, TTLoRA exhibits lower membership leakage even without DP training, using substantially smaller adapters and requiring on average 7.6X fewer parameters than LoRA. Overall, our results demonstrate that TTLoRA offers a practical path to improving the privacy-utility tradeoff in parameter-efficient language model adaptation.

Figures (7)

• Figure 1: Attack AUC 3D surface on Enron dataset (two views). LoRA shows significant variation in attack AUC (lower is better) across epsilon whereas TTLoRA maintains a relatively flat surface near the 50% random-guess baseline in private setting (non here means non-private setting).
• Figure 2: Calibrated loss distributions for MIA leakage (Enron) under non-private setting. Histograms show $\,\mathcal{L}(x;\theta_{\mathrm{peft}})-\mathcal{L}(x;\theta_{\mathrm{ref}})\,$ for members and non-members at ranks 2/4/6 for LoRA and TTLoRA; dashed line marks the 10% non-member quantile used for fixed-FPR evaluation. Larger separation implies higher AUROC and stronger leakage.
• Figure 3: TTLoRA Architecture: Forward Pass Tensor Contraction and Expansion with Input tensor. Red arrow indicates the contraction while green arrow indicates the expansion of the intermediate input activation
• Figure 4: Pareto Frontier: Best utility extracted among multiple configurations under differential privacy. TTLoRA achieves better perplexity than LoRA at all privacy budgets, with the advantage strongest at stricter (lower $\varepsilon$) settings.
• Figure 5: Overall utility comparison over different rank configurations for LoRA and TTLoRA along with FFT under DP training with Enron and PTB datasets