Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Almost Tight Error Bounds on Differentially Private Continual Counting

Monika Henzinger, Jalaj Upadhyay, Sarvagya Upadhyay

TL;DR

This work advances private continual counting by introducing a matrix-mechanism that factorizes the counting workload, achieving an almost exact mean-squared error bound with explicit constants and proving near-tight lower bounds for any DP mechanism. By bounding the γ_F(·) norm via an SDP and exploiting a two-factor (Toeplitz) factorization, the authors show an error improvement of roughly a factor of 10 over the traditional binary mechanism, while ensuring constant-time per-round releases. The results extend to private online learning through improved regret bounds and establish tight lower bounds for parity queries, highlighting the broad applicability of the γ_F framework to linear queries under differential privacy. Overall, the paper provides a principled, near-optimal blueprint for continual counting that is both theoretically tight and practically efficient, with clear implications for real-world private data deployments and online private learning.

Abstract

The first large-scale deployment of private federated learning uses differentially private counting in the continual release model as a subroutine (Google AI blog titled "Federated Learning with Formal Differential Privacy Guarantees"). In this case, a concrete bound on the error is very relevant to reduce the privacy parameter. The standard mechanism for continual counting is the binary mechanism. We present a novel mechanism and show that its mean squared error is both asymptotically optimal and a factor 10 smaller than the error of the binary mechanism. We also show that the constants in our analysis are almost tight by giving non-asymptotic lower and upper bounds that differ only in the constants of lower-order terms. Our algorithm is a matrix mechanism for the counting matrix and takes constant time per release. We also use our explicit factorization of the counting matrix to give an upper bound on the excess risk of the private learning algorithm of Denisov et al. (NeurIPS 2022). Our lower bound for any continual counting mechanism is the first tight lower bound on continual counting under approximate differential privacy. It is achieved using a new lower bound on a certain factorization norm, denoted by $γ_F(\cdot)$, in terms of the singular values of the matrix. In particular, we show that for any complex matrix, $A \in \mathbb{C}^{m \times n}$, \[ γ_F(A) \geq \frac{1}{\sqrt{m}}\|A\|_1, \] where $\|\cdot \|$ denotes the Schatten-1 norm. We believe this technique will be useful in proving lower bounds for a larger class of linear queries. To illustrate the power of this technique, we show the first lower bound on the mean squared error for answering parity queries.

Almost Tight Error Bounds on Differentially Private Continual Counting

TL;DR

This work advances private continual counting by introducing a matrix-mechanism that factorizes the counting workload, achieving an almost exact mean-squared error bound with explicit constants and proving near-tight lower bounds for any DP mechanism. By bounding the γ_F(·) norm via an SDP and exploiting a two-factor (Toeplitz) factorization, the authors show an error improvement of roughly a factor of 10 over the traditional binary mechanism, while ensuring constant-time per-round releases. The results extend to private online learning through improved regret bounds and establish tight lower bounds for parity queries, highlighting the broad applicability of the γ_F framework to linear queries under differential privacy. Overall, the paper provides a principled, near-optimal blueprint for continual counting that is both theoretically tight and practically efficient, with clear implications for real-world private data deployments and online private learning.

Abstract

The first large-scale deployment of private federated learning uses differentially private counting in the continual release model as a subroutine (Google AI blog titled "Federated Learning with Formal Differential Privacy Guarantees"). In this case, a concrete bound on the error is very relevant to reduce the privacy parameter. The standard mechanism for continual counting is the binary mechanism. We present a novel mechanism and show that its mean squared error is both asymptotically optimal and a factor 10 smaller than the error of the binary mechanism. We also show that the constants in our analysis are almost tight by giving non-asymptotic lower and upper bounds that differ only in the constants of lower-order terms. Our algorithm is a matrix mechanism for the counting matrix and takes constant time per release. We also use our explicit factorization of the counting matrix to give an upper bound on the excess risk of the private learning algorithm of Denisov et al. (NeurIPS 2022). Our lower bound for any continual counting mechanism is the first tight lower bound on continual counting under approximate differential privacy. It is achieved using a new lower bound on a certain factorization norm, denoted by , in terms of the singular values of the matrix. In particular, we show that for any complex matrix, , where denotes the Schatten-1 norm. We believe this technique will be useful in proving lower bounds for a larger class of linear queries. To illustrate the power of this technique, we show the first lower bound on the mean squared error for answering parity queries.
Paper Structure (46 sections, 28 theorems, 171 equations, 3 figures, 3 algorithms)

This paper contains 46 sections, 28 theorems, 171 equations, 3 figures, 3 algorithms.

Table of Contents

  1. Introduction
  2. Problem Statement and Our Contributions
  3. Main Result
  4. Other Contributions
  5. Contribution 1: An almost exact error bound for continual counting.
  6. Contribution 2: A lower bound on the $\ell_2^2$-error for any mechanism for continual counting.
  7. Contribution 3: Suboptimality of the binary mechanism.
  8. Contribution 4: Online Private Learning.
  9. Contribution 5: Lower Bounds on Special Linear Queries
  10. Our Techniques
  11. Main Result: Useful Properties of $\gamma_{\operatorname{F}}(\cdot)$
  12. Other Contributions
  13. Contribution 1: An almost exact error bound for continual counting.
  14. Upper bound in \ref{['eq:gammanormboud']}.
  15. Lower bound in \ref{['eq:gammanormboud']}.
  16. ...and 31 more sections

Key Result

Lemma 1

For a matrix $A\in\mathbb{C}^{n\times m}$, we have $\frac{{\left\Vert {A} \right\Vert}_1}{\sqrt{m}} \leqslant \gamma_{\operatorname{F}}(A) \leqslant {\left\Vert {A} \right\Vert}_{\operatorname{F}}$, where ${\left\Vert {A} \right\Vert}_1$ is the Schatten-$1$ norm (or, trace norm) of $A$. In particula

Figures (3)

  • Figure 1: Comparison of $\ell_2^2$-error bounds of binary and our factorization based mechanisms for varying $\varepsilon$ and $\delta = 10^{-10}$. Our mechanism ($\varepsilon=0.3$) incurs less error than binary mechanism ($\varepsilon=0.8$) for $n\geqslant 2^{19}$.
  • Figure 2: SDP for $\gamma_{\operatorname{F}}(\cdot)$ norm.
  • Figure :

Theorems & Definitions (62)

  • Lemma 1
  • Theorem 2
  • Theorem 3: Lower bound on matrix mechanisms.
  • Theorem 4: Lower bound on the $\ell_2^2$-error of continual counting
  • Theorem 5
  • Theorem 6
  • Remark 7
  • Definition 8: Parity Query
  • Theorem 9
  • Definition 10: Dual norm
  • ...and 52 more