Differential Privacy for Transformer Embeddings of Text with Nonparametric Variational Information Bottleneck

TL;DR

This work addresses privacy risks in sharing text data by sanitising transformer embeddings with a nonparametric variational information bottleneck (NVIB) to produce a stochastic bottleneck embedding. By sampling from the NVIB posterior and processing through a denoising attention block, the method achieves local differential privacy guided by Rényi divergence, with the privacy budget translated into $(\epsilon_\mu,\delta_\mu)$-BDP guarantees. The key contributions are (i) the NVIB-based, task-aware calibration of noise over multi-vector transformer embeddings, (ii) theoretical framing that links RD-based privacy accounting to BDPro guarantees, and (iii) empirical validation on GLUE showing strong privacy-utility trade-offs relative to task-agnostic baselines. The approach enables practical, privacy-preserving sharing of transformer embeddings in real-world NLP applications while maintaining competitive downstream performance.

Abstract

We propose a privacy-preserving method for sharing text data by sharing noisy versions of their transformer embeddings. It has been shown that hidden representations learned by deep models can encode sensitive information from the input, making it possible for adversaries to recover the input data with considerable accuracy. This problem is exacerbated in transformer embeddings because they consist of multiple vectors, one per token. To mitigate this risk, we propose Nonparametric Variational Differential Privacy (NVDP), which ensures both useful data sharing and strong privacy protection. We take a differential privacy approach, integrating a Nonparametric Variational Information Bottleneck (NVIB) layer into the transformer architecture to inject noise into its multi-vector embeddings and thereby hide information, and measuring privacy protection with Rényi divergence and its corresponding Bayesian Differential Privacy (BDP) guarantee. Training the NVIB layer calibrates the noise level according to utility. We test NVDP on the GLUE benchmark and show that varying the noise level gives us a useful tradeoff between privacy and accuracy. With lower noise levels, our model maintains high accuracy while offering strong privacy guarantees, effectively balancing privacy and utility.

Figures (3)

• Figure 1: An NVDP model. The model projects an input embedding into the parameters of a posterior distribution. A key modification for privacy is that we sample from this latent distribution during both training and testing. The resulting noisy representation is processed by a Denoising MHA layer. To enforce the privacy bottleneck, the standard residual skip connection around the MHA is removed, preventing any un-sanitized information from bypassing the bottleneck.
• Figure 2: Accuracy versus Bayesian Differential Privacy ($\epsilon_\mu$). The BDP budget ($\epsilon_\mu$) is calculated by finding the tightest privacy guarantee for a fixed $\delta_\mu=1e-5$. This provides a more interpretable view of the privacy-utility trade-off, where lower ($\epsilon_\mu$) values signify stronger, more practical privacy guarantees. The NVDP model consistently achieves better privacy-utility points than the VTDP ablation.
• Figure 3: Accuracy versus maximum Rényi Divergence (RD) illustrating the worst-case privacy-utility tradeoff. Each point corresponds to a different KL regularization weight, where stronger regularization leads to better privacy (lower RD). The most favorable models are those closest to the upper-right corner.

Theorems & Definitions (3)

• Definition 2.1
• Definition 2.2: $(\lambda, \epsilon)$-Rényi Differential Privacy
• Definition 2.3