Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Uncertainty quantification by block bootstrap for differentially private stochastic gradient descent

Holger Dette, Carina Graw

TL;DR

The paper tackles uncertainty quantification for SGD under local differential privacy by introducing a multiplier block bootstrap that accounts for the strong dependence in SGD iterates without requiring privacy budget splitting. It proves that, under appropriate martingale-difference and regularity conditions, the LDP-SGD estimator satisfies \\sqrt{n}(\\bar{\\theta}^{LDP}-\\theta_\\star) \\overset{d}{\\to} N(0,\\Sigma)$ with \\Sigma = G^{-1} S G^{-1}$, where $S$ aggregates both SGD and LDP noise and $G$ linearizes the gradient. The proposed bootstrap statistic \\bar{\\theta}^{\\star} mimics this limit distribution using block structure and random multipliers, achieving consistency and practical finite-sample performance with a fixed privacy budget. This yields a computationally tractable approach to UQ for LDP-SGD and, as a by-product, provides a simple UQ tool for non-private SGD. Simulations on quantile estimation and quantile regression demonstrate competitive coverage and interval lengths, underscoring the method's applicability to large-scale, privacy-preserving inference.

Abstract

Stochastic Gradient Descent (SGD) is a widely used tool in machine learning. In the context of Differential Privacy (DP), SGD has been well studied in the last years in which the focus is mainly on convergence rates and privacy guarantees. While in the non private case, uncertainty quantification (UQ) for SGD by bootstrap has been addressed by several authors, these procedures cannot be transferred to differential privacy due to multiple queries to the private data. In this paper, we propose a novel block bootstrap for SGD under local differential privacy that is computationally tractable and does not require an adjustment of the privacy budget. The method can be easily implemented and is applicable to a broad class of estimation problems. We prove the validity of our approach and illustrate its finite sample properties by means of a simulation study. As a by-product, the new method also provides a simple alternative numerical tool for UQ for non-private SGD.

Uncertainty quantification by block bootstrap for differentially private stochastic gradient descent

TL;DR

The paper tackles uncertainty quantification for SGD under local differential privacy by introducing a multiplier block bootstrap that accounts for the strong dependence in SGD iterates without requiring privacy budget splitting. It proves that, under appropriate martingale-difference and regularity conditions, the LDP-SGD estimator satisfies \\sqrt{n}(\\bar{\\theta}^{LDP}-\\theta_\\star) \\overset{d}{\\to} N(0,\\Sigma), where aggregates both SGD and LDP noise and linearizes the gradient. The proposed bootstrap statistic \\bar{\\theta}^{\\star} mimics this limit distribution using block structure and random multipliers, achieving consistency and practical finite-sample performance with a fixed privacy budget. This yields a computationally tractable approach to UQ for LDP-SGD and, as a by-product, provides a simple UQ tool for non-private SGD. Simulations on quantile estimation and quantile regression demonstrate competitive coverage and interval lengths, underscoring the method's applicability to large-scale, privacy-preserving inference.

Abstract

Stochastic Gradient Descent (SGD) is a widely used tool in machine learning. In the context of Differential Privacy (DP), SGD has been well studied in the last years in which the focus is mainly on convergence rates and privacy guarantees. While in the non private case, uncertainty quantification (UQ) for SGD by bootstrap has been addressed by several authors, these procedures cannot be transferred to differential privacy due to multiple queries to the private data. In this paper, we propose a novel block bootstrap for SGD under local differential privacy that is computationally tractable and does not require an adjustment of the privacy budget. The method can be easily implemented and is applicable to a broad class of estimation problems. We prove the validity of our approach and illustrate its finite sample properties by means of a simulation study. As a by-product, the new method also provides a simple alternative numerical tool for UQ for non-private SGD.
Paper Structure (19 sections, 12 theorems, 113 equations, 1 figure, 2 tables, 2 algorithms)

This paper contains 19 sections, 12 theorems, 113 equations, 1 figure, 2 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Preliminaries and Background
  3. Multiplier Block Bootstrap for LDP-SGD
  4. Simulation
  5. Technical assumptions
  6. Proofs of the results in Section \ref{['main']}
  7. Proof of Theorem \ref{['DP SGD normal nonlinear']}
  8. Proof of Theorem \ref{['BB is consistent nonlinear']}
  9. Proof of Theorem \ref{['BB is consistent linear']}
  10. Proofs of Lemma \ref{['decomposition bootstrap']} - Lemma \ref{['convergence in prop']}
  11. Proof of Lemma \ref{['decomposition bootstrap']}
  12. Proof of Lemma \ref{['convergence in distribution - new']}
  13. Proof of Lemma \ref{['convergence in prop']}
  14. Proofs of auxiliary results
  15. Proof of \ref{['E2 decomposition']}
  16. ...and 4 more sections

Key Result

Theorem 2.1

Figures (1)

  • Figure 1: Trajectory of the LDP-SGD estimate $\bar{\theta}_n^{LDP}$ (av. SGD) and the upper and lower boundaries of confidence intervals obtained by block bootstrap (BB), batch mean (BM) and self normalisation (SN) of the $50\%$ quantile of a standard normal distribution. The left and right panel correspond to different sample sizes, left: $n=10 - 200 000$; right: $n=200 000 - 1 000 000$.

Theorems & Definitions (22)

  • Definition 2.1
  • Theorem 2.1
  • Example 2.1: Laplace Mechanism
  • Example 2.2: Randomized Response
  • Remark 2.1
  • Remark 2.2
  • Theorem 3.1
  • Remark 3.1
  • Theorem 3.2
  • Remark 3.2
  • ...and 12 more