Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Differentially Private Random Block Coordinate Descent

Artavazd Maranjyan, Abdurakhmon Sadiev, Peter Richtárik

TL;DR

This work addresses privacy in high-dimensional optimization by proposing DP-SkGD, a differentially private random block coordinate descent that uses sketch matrices to update multiple coordinates per iteration. It generalizes both DP-CD and DP-SGD by enabling flexible block sampling and per-coordinate noise via a diagonal step matrix, and it provides rigorous privacy guarantees through Gaussian noise calibrated to coordinate subsets. The paper presents comprehensive utility bounds in convex and strongly convex settings, with refined rates for block sampling and importance sampling, and contrasts DP-SkGD with DP-SGD and DP-CD to reveal potential speedups of up to $\sqrt{d}$ under favorable problem structure. The approach leverages coordinate-wise smoothness constants and sampling strategies to achieve improved convergence while preserving differential privacy, offering a scalable and versatile framework for privacy-preserving optimization in high dimensions.

Abstract

Coordinate Descent (CD) methods have gained significant attention in machine learning due to their effectiveness in solving high-dimensional problems and their ability to decompose complex optimization tasks. However, classical CD methods were neither designed nor analyzed with data privacy in mind, a critical concern when handling sensitive information. This has led to the development of differentially private CD methods, such as DP-CD (Differentially Private Coordinate Descent) proposed by Mangold et al. (ICML 2022), yet a disparity remains between non-private CD and DP-CD methods. In our work, we propose a differentially private random block coordinate descent method that selects multiple coordinates with varying probabilities in each iteration using sketch matrices. Our algorithm generalizes both DP-CD and the classical DP-SGD (Differentially Private Stochastic Gradient Descent), while preserving the same utility guarantees. Furthermore, we demonstrate that better utility can be achieved through importance sampling, as our method takes advantage of the heterogeneity in coordinate-wise smoothness constants, leading to improved convergence rates.

Differentially Private Random Block Coordinate Descent

TL;DR

This work addresses privacy in high-dimensional optimization by proposing DP-SkGD, a differentially private random block coordinate descent that uses sketch matrices to update multiple coordinates per iteration. It generalizes both DP-CD and DP-SGD by enabling flexible block sampling and per-coordinate noise via a diagonal step matrix, and it provides rigorous privacy guarantees through Gaussian noise calibrated to coordinate subsets. The paper presents comprehensive utility bounds in convex and strongly convex settings, with refined rates for block sampling and importance sampling, and contrasts DP-SkGD with DP-SGD and DP-CD to reveal potential speedups of up to under favorable problem structure. The approach leverages coordinate-wise smoothness constants and sampling strategies to achieve improved convergence while preserving differential privacy, offering a scalable and versatile framework for privacy-preserving optimization in high dimensions.

Abstract

Coordinate Descent (CD) methods have gained significant attention in machine learning due to their effectiveness in solving high-dimensional problems and their ability to decompose complex optimization tasks. However, classical CD methods were neither designed nor analyzed with data privacy in mind, a critical concern when handling sensitive information. This has led to the development of differentially private CD methods, such as DP-CD (Differentially Private Coordinate Descent) proposed by Mangold et al. (ICML 2022), yet a disparity remains between non-private CD and DP-CD methods. In our work, we propose a differentially private random block coordinate descent method that selects multiple coordinates with varying probabilities in each iteration using sketch matrices. Our algorithm generalizes both DP-CD and the classical DP-SGD (Differentially Private Stochastic Gradient Descent), while preserving the same utility guarantees. Furthermore, we demonstrate that better utility can be achieved through importance sampling, as our method takes advantage of the heterogeneity in coordinate-wise smoothness constants, leading to improved convergence rates.

Paper Structure

This paper contains 29 sections, 19 theorems, 86 equations, 1 table, 2 algorithms.

Table of Contents

  1. Introduction
  2. Problem formulation and Contributions
  3. Summary of Contributions
  4. Preliminaries and Assumptions
  5. Norms
  6. Random Sets
  7. Sketch and Sparsification
  8. Assumptions
  9. Differential Privacy (DP)
  10. Differentially Private Sketched Gradient Descent
  11. Privacy Guarantees
  12. Utility Guarantees
  13. Block Sampling
  14. Comparison
  15. Comparing to DP-SGD
  16. ...and 14 more sections

Key Result

Lemma 3.9

Let $\ell: \mathbb{R}^d \times \mathcal{X} \rightarrow \mathbb{R}$ be differentiable in its first argument (ass:differentiability), convex (ass:convexity), and $L_{\mathcal{S}}$-component-Lipschitz with $L_U > 0$ for all $U \in \mathrm{Range}\left(\mathcal{S}\right)$ (ass:comp_lipschitz). Then for all $U \in \mathrm{Range}\left(\mathcal{S}\right)$.

Theorems & Definitions (35)

  • Definition 3.2: Unbiased diagonal sketch
  • Definition 3.8: Differential Privacy, dwork2008differential
  • Lemma 3.9: Proof in \ref{['section:proof_sensitivity']}
  • Theorem 4.1: Proof in \ref{['proof_privacy']}
  • Theorem 4.2: Proof in \ref{['proof_utility']}
  • Lemma C.1
  • proof
  • Lemma \ref{lem:sensitivity}
  • proof
  • Definition D.1: Rényi divergence renyi1961measuresvan2014renyi
  • ...and 25 more