Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Nonparametric Variational Differential Privacy via Embedding Parameter Clipping

Dina El Zein, Shashi Kumar, James Henderson

TL;DR

This work presents a simple yet effective method for improving the privacy-utility trade-off in variational models, making them more robust and practical.

Abstract

The nonparametric variational information bottleneck (NVIB) provides the foundation for nonparametric variational differential privacy (NVDP), a framework for building privacy-preserving language models. However, the learned latent representations can drift into regions with high information content, leading to poor privacy guarantees, but also low utility due to numerical instability during training. In this work, we introduce a principled parameter clipping strategy to directly address this issue. Our method is mathematically derived from the objective of minimizing the Rényi Divergence (RD) upper bound, yielding specific, theoretically grounded constraints on the posterior mean, variance, and mixture weight parameters. We apply our technique to an NVIB based model and empirically compare it against an unconstrained baseline. Our findings demonstrate that the clipped model consistently achieves tighter RD bounds, implying stronger privacy, while simultaneously attaining higher performance on several downstream tasks. This work presents a simple yet effective method for improving the privacy-utility trade-off in variational models, making them more robust and practical.

Nonparametric Variational Differential Privacy via Embedding Parameter Clipping

TL;DR

This work presents a simple yet effective method for improving the privacy-utility trade-off in variational models, making them more robust and practical.

Abstract

The nonparametric variational information bottleneck (NVIB) provides the foundation for nonparametric variational differential privacy (NVDP), a framework for building privacy-preserving language models. However, the learned latent representations can drift into regions with high information content, leading to poor privacy guarantees, but also low utility due to numerical instability during training. In this work, we introduce a principled parameter clipping strategy to directly address this issue. Our method is mathematically derived from the objective of minimizing the Rényi Divergence (RD) upper bound, yielding specific, theoretically grounded constraints on the posterior mean, variance, and mixture weight parameters. We apply our technique to an NVIB based model and empirically compare it against an unconstrained baseline. Our findings demonstrate that the clipped model consistently achieves tighter RD bounds, implying stronger privacy, while simultaneously attaining higher performance on several downstream tasks. This work presents a simple yet effective method for improving the privacy-utility trade-off in variational models, making them more robust and practical.
Paper Structure (34 sections, 15 equations, 1 figure, 7 tables)

This paper contains 34 sections, 15 equations, 1 figure, 7 tables.

Table of Contents

  1. Introduction
  2. Background
  3. Rényi Differential Privacy (RDP)
  4. Bayesian Differential Privacy (BDP)
  5. Nonparametric Variational Differential Privacy (NVDP)
  6. Principled Clipping for Tighter Rényi Divergence Bounds
  7. The Rényi Divergence Bound
  8. Derivation of Clipping Operations
  9. Mean Clipping ($\boldsymbol{\mu}_i^q$).
  10. Standard Deviation Clipping ($\boldsymbol{\sigma}_i^q$).
  11. Alpha Clipping ($\alpha_i^q$).
  12. Experiments
  13. Evaluation Metrics and Privacy Accounting
  14. Natural Language Understanding Tasks
  15. Datasets.
  16. ...and 19 more sections

Figures (1)

  • Figure 1: The Nonparametric Variational Differential Privacy (NVDP) Architecture. An input sequence is first processed by a standard pretrained Transformer encoder to produce token-wise embeddings, $\boldsymbol{x} \in \mathbb{R}^{n\times d}$. These embeddings are then passed through a NVIB layer, which acts as a stochastic bottleneck by mapping them to the parameters of a posterior distribution. A sample is drawn from this distribution and processed by a Denoising Multi-Head Attention (MHA) and a Feed-Forward layer. Crucially, the standard residual skip connection around the MHA and Feed-Forward block is removed, ensuring that all information must pass through the privacy-preserving NVIB layer.

Theorems & Definitions (2)

  • Definition 2.1: Rényi Divergence
  • Definition 2.2: Bayesian Differential Privacy