Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Distributed Online Private Learning of Convex Nondecomposable Objectives

Huqiang Cheng, Xiaofeng Liao, Huaqing Li

TL;DR

The paper tackles distributed online learning with privacy for convex nondecomposable objectives over time-varying networks. It introduces DPSDA, a differentially private stochastic dual-averaging framework, and two concrete instantiations: DPSDA-C for undirected networks and DPSDA-PS for directed networks, both attaining $\mathcal{O}(\sqrt{T})$ expected regret under convexity. A key contribution is the explicit quantification of the privacy-utility trade-off via Laplace perturbations, including sensitivity bounds and horizon-wide privacy guarantees. Theoretical results are complemented by numerical experiments on synthetic and real datasets, confirming the effectiveness of the methods and the impact of privacy levels on performance, with practical implications for privacy-preserving distributed learning in dynamic networks.

Abstract

We deal with a general distributed constrained online learning problem with privacy over time-varying networks, where a class of nondecomposable objectives are considered. Under this setting, each node only controls a part of the global decision, and the goal of all nodes is to collaboratively minimize the global cost over a time horizon $T$ while guarantees the security of the transmitted information. For such problems, we first design a novel generic algorithm framework, named as DPSDA, of differentially private distributed online learning using the Laplace mechanism and the stochastic variants of dual averaging method. Note that in the dual updates, all nodes of DPSDA employ the noise-corrupted gradients for more generality. Then, we propose two algorithms, named as DPSDA-C and DPSDA-PS, under this framework. In DPSDA-C, the nodes implement a circulation-based communication in the primal updates so as to alleviate the disagreements over time-varying undirected networks. In addition, for the extension to time-varying directed ones, the nodes implement the broadcast-based push-sum dynamics in DPSDA-PS, which can achieve average consensus over arbitrary directed networks. Theoretical results show that both algorithms attain an expected regret upper bound in $\mathcal{O}( \sqrt{T} )$ when the objective function is convex, which matches the best utility achievable by cutting-edge algorithms. Finally, numerical experiment results on both synthetic and real-world datasets verify the effectiveness of our algorithms.

Distributed Online Private Learning of Convex Nondecomposable Objectives

TL;DR

The paper tackles distributed online learning with privacy for convex nondecomposable objectives over time-varying networks. It introduces DPSDA, a differentially private stochastic dual-averaging framework, and two concrete instantiations: DPSDA-C for undirected networks and DPSDA-PS for directed networks, both attaining expected regret under convexity. A key contribution is the explicit quantification of the privacy-utility trade-off via Laplace perturbations, including sensitivity bounds and horizon-wide privacy guarantees. Theoretical results are complemented by numerical experiments on synthetic and real datasets, confirming the effectiveness of the methods and the impact of privacy levels on performance, with practical implications for privacy-preserving distributed learning in dynamic networks.

Abstract

We deal with a general distributed constrained online learning problem with privacy over time-varying networks, where a class of nondecomposable objectives are considered. Under this setting, each node only controls a part of the global decision, and the goal of all nodes is to collaboratively minimize the global cost over a time horizon while guarantees the security of the transmitted information. For such problems, we first design a novel generic algorithm framework, named as DPSDA, of differentially private distributed online learning using the Laplace mechanism and the stochastic variants of dual averaging method. Note that in the dual updates, all nodes of DPSDA employ the noise-corrupted gradients for more generality. Then, we propose two algorithms, named as DPSDA-C and DPSDA-PS, under this framework. In DPSDA-C, the nodes implement a circulation-based communication in the primal updates so as to alleviate the disagreements over time-varying undirected networks. In addition, for the extension to time-varying directed ones, the nodes implement the broadcast-based push-sum dynamics in DPSDA-PS, which can achieve average consensus over arbitrary directed networks. Theoretical results show that both algorithms attain an expected regret upper bound in when the objective function is convex, which matches the best utility achievable by cutting-edge algorithms. Finally, numerical experiment results on both synthetic and real-world datasets verify the effectiveness of our algorithms.
Paper Structure (38 sections, 14 theorems, 75 equations, 8 figures, 5 tables, 3 algorithms)

This paper contains 38 sections, 14 theorems, 75 equations, 8 figures, 5 tables, 3 algorithms.

Table of Contents

  1. Introduction
  2. Related Works
  3. Contributions
  4. Organization and Notations
  5. Preliminaries
  6. Graph Theory
  7. Problem Formulation
  8. Privacy Concern and Attack Model
  9. Differential Privacy
  10. The Basic Framework and Regret Bound
  11. The Basic Framework---DPSDA
  12. The Generic Regret Bound
  13. DPSDA-C AND ITS ANALYSIS
  14. DPSDA-C
  15. Differential Privacy Analysis of DPSDA-C
  16. ...and 23 more sections

Key Result

Proposition 1

(Duchi2012) Given a sequence of random vectors $\left\{ \boldsymbol{\varphi }\left( t \right) \right\} _{t\in \mathbb{Z}_+}\subseteq \mathbb{R}^n$, we define a network-level summation of random variables by Then, it follows that, for $T\in \mathbb{Z}_+$ and $\mathbf{y}\in \chi ^n$, In addition, for any $\mathbf{z}_1,\mathbf{z}_2\in \mathbb{R}^n$, it holds

Figures (8)

  • Figure 1: The behaviors of the decision variable in nondecomposable problems (a) and decomposable problems (b).
  • Figure 2: A $4$-strongly connect time-varying directed communication topology.
  • Figure 3: Performance on Synthetic data over different $\epsilon$.
  • Figure 4: MNIST (Left) and CIFAR-10 (Right) dataset.
  • Figure 5: Performance on real datasets over different $\epsilon$.
  • ...and 3 more figures

Theorems & Definitions (41)

  • Remark 1
  • Definition 1
  • Definition 2
  • Definition 3
  • Remark 2
  • Remark 3
  • Proposition 1
  • Theorem 1
  • proof
  • Remark 4
  • ...and 31 more