Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Noise-Aware Differentially Private Regression via Meta-Learning

Ossi Räisä, Stratis Markou, Matthew Ashman, Wessel P. Bruinsma, Marlon Tobaben, Antti Honkela, Richard E. Turner

TL;DR

The paper introduces DPConvCNP, a meta-learned regression model that embeds a differential privacy mechanism inside a ConvCNP via a functional DP mechanism, enabling noise-aware predictions with calibrated uncertainty in small-data regimes. It tightens functional-DP bounds using Gaussian DP theory and integrates a DPSetConv encoder to protect context data, delivering $(\epsilon,\delta)$-DP guarantees for meta-testing. Empirically, DPConvCNP matches or surpasses a carefully tuned DP Gaussian Process baseline on Gaussian and non-Gaussian tasks while being significantly faster at test time, and demonstrates strong sim-to-real performance under privacy constraints. The work also enables amortisation over privacy budgets and shows robust calibration for real-world, privacy-preserving regression tasks, with limitations including modeling dependencies among targets and reliance on simulator diversity for sim-to-real transfer.

Abstract

Many high-stakes applications require machine learning models that protect user privacy and provide well-calibrated, accurate predictions. While Differential Privacy (DP) is the gold standard for protecting user privacy, standard DP mechanisms typically significantly impair performance. One approach to mitigating this issue is pre-training models on simulated data before DP learning on the private data. In this work we go a step further, using simulated data to train a meta-learning model that combines the Convolutional Conditional Neural Process (ConvCNP) with an improved functional DP mechanism of Hall et al. [2013] yielding the DPConvCNP. DPConvCNP learns from simulated data how to map private data to a DP predictive model in one forward pass, and then provides accurate, well-calibrated predictions. We compare DPConvCNP with a DP Gaussian Process (GP) baseline with carefully tuned hyperparameters. The DPConvCNP outperforms the GP baseline, especially on non-Gaussian data, yet is much faster at test time and requires less tuning.

Noise-Aware Differentially Private Regression via Meta-Learning

TL;DR

The paper introduces DPConvCNP, a meta-learned regression model that embeds a differential privacy mechanism inside a ConvCNP via a functional DP mechanism, enabling noise-aware predictions with calibrated uncertainty in small-data regimes. It tightens functional-DP bounds using Gaussian DP theory and integrates a DPSetConv encoder to protect context data, delivering -DP guarantees for meta-testing. Empirically, DPConvCNP matches or surpasses a carefully tuned DP Gaussian Process baseline on Gaussian and non-Gaussian tasks while being significantly faster at test time, and demonstrates strong sim-to-real performance under privacy constraints. The work also enables amortisation over privacy budgets and shows robust calibration for real-world, privacy-preserving regression tasks, with limitations including modeling dependencies among targets and reliance on simulator diversity for sim-to-real transfer.

Abstract

Many high-stakes applications require machine learning models that protect user privacy and provide well-calibrated, accurate predictions. While Differential Privacy (DP) is the gold standard for protecting user privacy, standard DP mechanisms typically significantly impair performance. One approach to mitigating this issue is pre-training models on simulated data before DP learning on the private data. In this work we go a step further, using simulated data to train a meta-learning model that combines the Convolutional Conditional Neural Process (ConvCNP) with an improved functional DP mechanism of Hall et al. [2013] yielding the DPConvCNP. DPConvCNP learns from simulated data how to map private data to a DP predictive model in one forward pass, and then provides accurate, well-calibrated predictions. We compare DPConvCNP with a DP Gaussian Process (GP) baseline with carefully tuned hyperparameters. The DPConvCNP outperforms the GP baseline, especially on non-Gaussian data, yet is much faster at test time and requires less tuning.
Paper Structure (32 sections, 12 theorems, 50 equations, 16 figures, 1 table, 4 algorithms)

This paper contains 32 sections, 12 theorems, 50 equations, 16 figures, 1 table, 4 algorithms.

Table of Contents

  1. Introduction
  2. Related Work
  3. Background
  4. Meta-learning and Neural Processes
  5. Differential Privacy
  6. The Functional Mechanism
  7. Differential privacy for the ConvCNP
  8. Improving the Functional Mechanism
  9. Differentially Private Convolutional CNP
  10. Training the DPConvCNP
  11. Experiments & Discussion
  12. Synthetic tasks
  13. Sim-to-real tasks
  14. Limitations & Conclusion
  15. Differential Privacy Details
  16. ...and 17 more sections

Key Result

Theorem 3.3

Let $\mathcal{M}$ be an $(\epsilon, \delta)$-DP (or $\mu$-GDP) algorithm and let $f$ be any, possibly randomised, function. Then $f\circ \mathcal{M}$ is $(\epsilon, \delta)$-DP (or $\mu$-GDP).

Figures (16)

  • Figure 1: Meta-training (left) and meta-testing (right) using our method. We train a model on multiple tasks with non-private (simulated or proxy) data to predict on target $(t)$ points using the context $(c)$ points. Crucially, by including a DP mechanism, which clips and adds noise to the data during training, the parameter updates (dashed arrow) teach the model to make well-calibrated and accurate predictions in the presence of DP noise. At test time, we deploy the model on real data using the same mechanism, which protects the context set with DP guarantees.
  • Figure 2: Training our proposed model with a DP mechanism inside it, enables the model to make accurate well-calibrated predictions, even for modest privacy budgets and dataset sizes. Here, the context data (black) are protected with different $(\epsilon, \delta)$ DP budgets as indicated. The model makes predictions (blue) that are remarkably close to the optimal (non-private) Bayes predictor.
  • Figure 3: Left; Illustration of the ConvCNP encoder $\texttt{enc}_\phi.$ Black crosses show an example context set $D^{(c)}.$ The density channel $r^{(d)}$ is shown in purple and the signal channel $r^{(r)}$ is shown in red. The representation $r$ consists of concatenating $r^{(d)}$ and $r^{(s)}.$ Right; Illustration of the DPConvCNP encoder. Black crosses show an example context $D^{(c)},$ clipped with a threshold $C$ (gray dashed). Here, a single point (rightmost) is clipped (gray cross shows value before clipping). The density and signal channels are computed and GP noise is added to obtain the DP representation (red & purple).
  • Figure 4: Noise magnitude comparison for the classical functional mechanism of hallDifferentialPrivacyFunctions2013, the RDP-based mechanism of jiangFunctionalRenyiDifferential2023 and our improved GDP-based mechanism. The line for hallDifferentialPrivacyFunctions2013 cuts off at $\epsilon = 1$ since their bound has only been proven for $\epsilon \leq 1$. We set $\Delta^2 = 10$ and $\delta = 10^{-3}$, which are representative values from our experiments. See Appendix \ref{['app:accountant-comparison-details']} for more details.
  • Figure 5: Deployment-time comparison on Gaussian (top) and non-Gaussian (bottom) data. We ran the DP-SVGP for different numbers of DP-SGD steps to determine a speed versus quality-of-fit tradeoff. Reporting 95% confidence intervals.
  • ...and 11 more figures

Theorems & Definitions (24)

  • Definition 3.1
  • Definition 3.2
  • Theorem 3.3: dworkAlgorithmicFoundationsDifferential2014
  • Theorem 3.4: dongGaussianDifferentialPrivacy2022
  • Theorem 3.5: dongGaussianDifferentialPrivacy2022
  • Definition 3.6
  • Theorem 3.7: hallDifferentialPrivacyFunctions2013
  • Theorem 4.1
  • proof
  • Theorem 4.2
  • ...and 14 more