Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Matrix Factorization for Practical Continual Mean Estimation Under User-Level Differential Privacy

Nikita P. Kalinin, Ali Najar, Valentin Roth, Christoph H. Lampert

TL;DR

This work addresses continual mean estimation under user-level differential privacy by leveraging a Matrix Factorization mechanism. It introduces a mean-estimation–oriented factorization, notably using a mean-aware lower-triangular Toeplitz correlation matrix $\mathbf{D}_{\mathrm{Toep}}$ (and its banded inverse $\mathbf{D}_{\mathrm{Toep}}^{p}$) to reduce noise under approximate DP, and provides RMSE and concentration bounds for both single and multi-participation. Theoretical results show asymptotically tighter RMSE and improved early-time performance compared to baselines, with empirical validation on real data demonstrating practical gains over prior pure-DP approaches. The paper also extends the analysis to distributional assumptions with sub-Gaussian data and discusses combining Matrix Factorization with withholding schemes to further improve utility in the approximate DP setting.

Abstract

We study continual mean estimation, where data vectors arrive sequentially and the goal is to maintain accurate estimates of the running mean. We address this problem under user-level differential privacy, which protects each user's entire dataset even when they contribute multiple data points. Previous work on this problem has focused on pure differential privacy. While important, this approach limits applicability, as it leads to overly noisy estimates. In contrast, we analyze the problem under approximate differential privacy, adopting recent advances in the Matrix Factorization mechanism. We introduce a novel mean estimation specific factorization, which is both efficient and accurate, achieving asymptotically lower mean-squared error bounds in continual mean estimation under user-level differential privacy.

Matrix Factorization for Practical Continual Mean Estimation Under User-Level Differential Privacy

TL;DR

This work addresses continual mean estimation under user-level differential privacy by leveraging a Matrix Factorization mechanism. It introduces a mean-estimation–oriented factorization, notably using a mean-aware lower-triangular Toeplitz correlation matrix (and its banded inverse ) to reduce noise under approximate DP, and provides RMSE and concentration bounds for both single and multi-participation. Theoretical results show asymptotically tighter RMSE and improved early-time performance compared to baselines, with empirical validation on real data demonstrating practical gains over prior pure-DP approaches. The paper also extends the analysis to distributional assumptions with sub-Gaussian data and discusses combining Matrix Factorization with withholding schemes to further improve utility in the approximate DP setting.

Abstract

We study continual mean estimation, where data vectors arrive sequentially and the goal is to maintain accurate estimates of the running mean. We address this problem under user-level differential privacy, which protects each user's entire dataset even when they contribute multiple data points. Previous work on this problem has focused on pure differential privacy. While important, this approach limits applicability, as it leads to overly noisy estimates. In contrast, we analyze the problem under approximate differential privacy, adopting recent advances in the Matrix Factorization mechanism. We introduce a novel mean estimation specific factorization, which is both efficient and accurate, achieving asymptotically lower mean-squared error bounds in continual mean estimation under user-level differential privacy.
Paper Structure (18 sections, 60 theorems, 257 equations, 4 figures, 3 tables, 4 algorithms)

This paper contains 18 sections, 60 theorems, 257 equations, 4 figures, 3 tables, 4 algorithms.

Table of Contents

  1. Introduction
  2. Background
  3. Private User Level Mean Estimation with Matrix Factorization
  4. Single Participation
  5. Multi-participation
  6. Experiments
  7. Distributional Assumptions
  8. Conclusion and Future Directions
  9. Concentration Bounds for the Mean
  10. Key Relationships
  11. Proofs
  12. Single Participation
  13. Multi-Participation
  14. Exponential Withhold-Release
  15. Diversity Conditions
  16. ...and 3 more sections

Key Result

Lemma 1

Let $f:\mathcal{X}^n \to \mathbb{R}^d$ have $\ell_2$-sensitivity $\Delta_2(f) := \sup_{x\sim x'} \|f(x)-f(x')\|_2$, where $x$ and $x'$ differ in one individual's data. The mechanism satisfies $(\varepsilon,\delta)$-differential privacy for $\varepsilon,\delta\in(0,1)$ if

Figures (4)

  • Figure 1: RMSE at step $n$ of different factorizations in the single-participation (item-level privacy) setting, presenting the error ratios of the four best-performing factorizations relative to BandMF mckenna2024scaling.
  • Figure 2: RMSE at step $n$ of our proposed factorization and the prefix sum based factorization divided by the RMSE of the banded $\mathbf{D}_{\mathrm{Toep}}$ factorization, in the multi-participation setting. The bandwidth for $\mathbf{E}_1^{1/2}$ is set to $p=\lceil\log_2 b\rceil$ and for $\mathbf{D}_{\mathrm{Toep}}$ it is set to $p= b$. The banded version of $\mathbf{D}_{\mathrm{Toep}}$ shows a slight benefit over the proposed banded inverse, with the difference diminishing as the matrix size grows.
  • Figure 3: The error of mean estimation over time for $\varepsilon=1$ and $\delta=10^{-6}$ with $k = 8$ participations measured in root averaged mean squared error. We used the banded inverse versions of $\mathbf{E}_1^{1/2}$ and $\mathbf{D}_{\mathrm{Toep}}$ with $p=16$ and $\mathbf{E}_{\nu}$ with $\nu=0.5$. As a baseline, we use the procedure from george2024continual for Continual Mean Estimation (CME).
  • Figure 4: The error of mean estimation for credit card transactions dataset for $\delta=5\times 10^{-6}$, and $\varepsilon=10$. We set $b=500$ and clip the stream of data by $\xi = 1000$ for the private mean.

Theorems & Definitions (98)

  • Lemma 1: dwork2014algorithmic
  • Theorem 1: Theorem 2 from kalinin2024banded
  • Definition 1: Mean-Aware Matrix Factorization
  • Lemma 2
  • Theorem 2
  • Theorem 3: RMSE Lower Bound
  • Lemma 3
  • Corollary 1
  • Lemma 4
  • Theorem 4
  • ...and 88 more