Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

On the Growth of Mistakes in Differentially Private Online Learning: A Lower Bound Perspective

Daniil Dmitriev, Kristóf Szabó, Amartya Sanyal

TL;DR

This work shows that, for a broad class of $(\varepsilon,\delta)-DP online algorithms, for number of rounds $T$ such that $\log T\leq O(1 / \delta)$, the expected number of mistakes incurred by the algorithm grows as $\Omega(\log T}{\delta})$.

Abstract

In this paper, we provide lower bounds for Differentially Private (DP) Online Learning algorithms. Our result shows that, for a broad class of $(\varepsilon,δ)$-DP online algorithms, for number of rounds $T$ such that $\log T\leq O(1 / δ)$, the expected number of mistakes incurred by the algorithm grows as $Ω(\log \frac{T}δ)$. This matches the upper bound obtained by Golowich and Livni (2021) and is in contrast to non-private online learning where the number of mistakes is independent of $T$. To the best of our knowledge, our work is the first result towards settling lower bounds for DP-Online learning and partially addresses the open question in Sanyal and Ramponi (2022).

On the Growth of Mistakes in Differentially Private Online Learning: A Lower Bound Perspective

TL;DR

This work shows that, for a broad class of T\log T\leq O(1 / \delta)\Omega(\log T}{\delta})$.

Abstract

In this paper, we provide lower bounds for Differentially Private (DP) Online Learning algorithms. Our result shows that, for a broad class of -DP online algorithms, for number of rounds such that , the expected number of mistakes incurred by the algorithm grows as . This matches the upper bound obtained by Golowich and Livni (2021) and is in contrast to non-private online learning where the number of mistakes is independent of . To the best of our knowledge, our work is the first result towards settling lower bounds for DP-Online learning and partially addresses the open question in Sanyal and Ramponi (2022).
Paper Structure (21 sections, 7 theorems, 28 equations, 1 figure)

This paper contains 21 sections, 7 theorems, 28 equations, 1 figure.

Table of Contents

  1. Introduction
  2. Main Result
  3. Preliminaries
  4. Online Learning
  5. Littlestone dimension
  6. Differential Privacy
  7. Related work
  8. Lower Bound for Private Online learning under Concentration assumption
  9. Main Result
  10. Examples of $\operatorname{\beta-concentrated}$ online learners for $\mathrm{Point}_N$
  11. Discussion
  12. Connection to under continual observation
  13. Beyond concentration assumption
  14. Pure differentially private online learners
  15. Open problems
  16. ...and 6 more sections

Key Result

Corollary 1

There exists a hypothesis class $\mathcal{H}$ with $\mathrm{Ldim}\left({\mathcal{H}}\right)=1$ (see defn:littlestone), such that for any $\varepsilon, \delta > 0, T \leqslant \exp(1/(32\delta))$, and any online learner $\mathcal{A}$ that is $(\varepsilon, \delta)$- and $0.1$-concentrated, there is a where $\widetilde{\Omega}$ hides logarithmic factors in $\varepsilon$. For $T > \exp(1/(32\delta))$

Figures (1)

  • Figure 1: Lower bound from \ref{['thm:main-finite']} vs. existing upper bounds. We assume that $\varepsilon, \delta$ are fixed and for simplicity ignore dependence on $\varepsilon$. X-axis corresponds to the number of samples, growing from $T \sim 1/\delta$ to $T \sim \exp(1/\delta)$ and larger. Y-axis shows the expected number of mistakes, $\mathop{\mathrm{\mathbb{E}}}\limits \left[{M_{\mathcal{A}}}\right]$.

Theorems & Definitions (17)

  • Corollary 1: Informal Corollary of \ref{['thm:main-finite']}
  • Definition 1: General game
  • Definition 2
  • Definition 3: Proper and Improper learner
  • Definition 4: Point class
  • Definition 5: Approximate differential privacy
  • Definition 6
  • Definition 7
  • Theorem 1
  • Definition 8: Multi-Point class
  • ...and 7 more