Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Tight Differentially Private PCA via Matrix Coherence

Tommaso d'Orsi, Gleb Novikov

TL;DR

It is proved that (rank-$r$) coherence does not increase under Gaussian perturbations, which implies that any estimator based on the Gaussian mechanism -- including the authors' -- preserves the coherence of the input.

Abstract

We revisit the task of computing the span of the top $r$ singular vectors $u_1, \ldots, u_r$ of a matrix under differential privacy. We show that a simple and efficient algorithm -- based on singular value decomposition and standard perturbation mechanisms -- returns a private rank-$r$ approximation whose error depends only on the \emph{rank-$r$ coherence} of $u_1, \ldots, u_r$ and the spectral gap $σ_r - σ_{r+1}$. This resolves a question posed by Hardt and Roth~\cite{hardt2013beyond}. Our estimator outperforms the state of the art -- significantly so in some regimes. In particular, we show that in the dense setting, it achieves the same guarantees for single-spike PCA in the Wishart model as those attained by optimal non-private algorithms, whereas prior private algorithms failed to do so. In addition, we prove that (rank-$r$) coherence does not increase under Gaussian perturbations. This implies that any estimator based on the Gaussian mechanism -- including ours -- preserves the coherence of the input. We conjecture that similar behavior holds for other structured models, including planted problems in graphs. We also explore applications of coherence to graph problems. In particular, we present a differentially private algorithm for Max-Cut and other constraint satisfaction problems under low coherence assumptions.

Tight Differentially Private PCA via Matrix Coherence

TL;DR

It is proved that (rank-) coherence does not increase under Gaussian perturbations, which implies that any estimator based on the Gaussian mechanism -- including the authors' -- preserves the coherence of the input.

Abstract

We revisit the task of computing the span of the top singular vectors of a matrix under differential privacy. We show that a simple and efficient algorithm -- based on singular value decomposition and standard perturbation mechanisms -- returns a private rank- approximation whose error depends only on the \emph{rank- coherence} of and the spectral gap . This resolves a question posed by Hardt and Roth~\cite{hardt2013beyond}. Our estimator outperforms the state of the art -- significantly so in some regimes. In particular, we show that in the dense setting, it achieves the same guarantees for single-spike PCA in the Wishart model as those attained by optimal non-private algorithms, whereas prior private algorithms failed to do so. In addition, we prove that (rank-) coherence does not increase under Gaussian perturbations. This implies that any estimator based on the Gaussian mechanism -- including ours -- preserves the coherence of the input. We conjecture that similar behavior holds for other structured models, including planted problems in graphs. We also explore applications of coherence to graph problems. In particular, we present a differentially private algorithm for Max-Cut and other constraint satisfaction problems under low coherence assumptions.

Paper Structure

This paper contains 43 sections, 32 theorems, 171 equations, 8 algorithms.

Table of Contents

  1. Introduction
  2. Main result
  3. Coherence of our estimator.
  4. Coherence of our estimator.
  5. Coherence of graphs under random perturbations
  6. Differentially private CSP solvers
  7. Organization
  8. Techniques
  9. Approximating the top singular space
  10. Coherence of our estimator.
  11. Private CSP solvers via differentially private PCA
  12. Future work
  13. Preliminaries
  14. Differential privacy
  15. Differentialy private low rank matrix estimation
  16. ...and 28 more sections

Key Result

Theorem 1.3

Let $M\in\varmathbb R^{n\times m}$, and let $r, r'\in [\mathop{\mathrm{rank}}\nolimits(M)]$ such that $r'\geqslant r$. Let $U_{(r)} \in \varmathbb R^{n\times r}$ be the matrix whose columns are the top $r$ left singular vectors of $M$. There exists an efficient, $(\varepsilon,\delta)$-differentially where $L = \frac{\sigma_r \log (n+m) }{\sigma_r - \sigma_{r+1}}$ .

Theorems & Definitions (73)

  • Definition 1.1: Basic coherence
  • Definition 1.2: Closeness of subspaces
  • Theorem 1.3: hardt2014noisy
  • Definition 1.4: $r$-Coherence
  • Theorem 1.5: Private singular subspace estimator
  • Corollary 1.6: Private eigenspace estimator
  • Conjecture 1.7
  • Theorem 1.8: Edge-DP max cut for low coherence graphs, simplified
  • Definition 4.1: $\tau$-threshold rank
  • Definition 4.2: Matrix adjacency
  • ...and 63 more