Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Generalizing Differentially Private Decentralized Deep Learning with Multi-Agent Consensus

Jasmine Bayrooti, Zhan Gao, Amanda Prorok

TL;DR

This work proposes a framework that embeds differential privacy into decentralized deep learning and secures each agent's local dataset during and after cooperative training and proves convergence guarantees for algorithms derived from this framework and its practical utility when applied to subgradient and ADMM decentralized approaches.

Abstract

Cooperative decentralized learning relies on direct information exchange between communicating agents, each with access to locally available datasets. The goal is to agree on model parameters that are optimal over all data. However, sharing parameters with untrustworthy neighbors can incur privacy risks by leaking exploitable information. To enable trustworthy cooperative learning, we propose a framework that embeds differential privacy into decentralized deep learning and secures each agent's local dataset during and after cooperative training. We prove convergence guarantees for algorithms derived from this framework and demonstrate its practical utility when applied to subgradient and ADMM decentralized approaches, finding accuracies approaching the centralized baseline while ensuring individual data samples are resilient to inference attacks. Furthermore, we study the relationships between accuracy, privacy budget, and networks' graph properties on collaborative classification tasks, discovering a useful invariance to the communication graph structure beyond a threshold.

Generalizing Differentially Private Decentralized Deep Learning with Multi-Agent Consensus

TL;DR

This work proposes a framework that embeds differential privacy into decentralized deep learning and secures each agent's local dataset during and after cooperative training and proves convergence guarantees for algorithms derived from this framework and its practical utility when applied to subgradient and ADMM decentralized approaches.

Abstract

Cooperative decentralized learning relies on direct information exchange between communicating agents, each with access to locally available datasets. The goal is to agree on model parameters that are optimal over all data. However, sharing parameters with untrustworthy neighbors can incur privacy risks by leaking exploitable information. To enable trustworthy cooperative learning, we propose a framework that embeds differential privacy into decentralized deep learning and secures each agent's local dataset during and after cooperative training. We prove convergence guarantees for algorithms derived from this framework and demonstrate its practical utility when applied to subgradient and ADMM decentralized approaches, finding accuracies approaching the centralized baseline while ensuring individual data samples are resilient to inference attacks. Furthermore, we study the relationships between accuracy, privacy budget, and networks' graph properties on collaborative classification tasks, discovering a useful invariance to the communication graph structure beyond a threshold.
Paper Structure (9 sections, 1 theorem, 7 equations, 1 algorithm)

This paper contains 9 sections, 1 theorem, 7 equations, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Problem Statement
  4. Background
  5. Differential Privacy
  6. Decentralized Optimization
  7. General Decentralized Differentially Private Learning
  8. Framework for Decentralized Differentially Private Learning
  9. Convergence Analysis

Key Result

theorem thmcountertheorem

Assume that $M(\theta)$ of the base decentralized algorithm is Lipschitz w.r.t. a constant $C_L$, i.e., $\|M(\theta) - M(\hat{\theta})\|_2 \le C_L \|\theta - \hat{\theta}\|_2$ for any $\theta$ and $\hat{\theta}$. If $M(\theta)$ of the base decentralized algorithm converges to the optimal solution $M

Theorems & Definitions (3)

  • definition thmcounterdefinition: Adjacent Datasets 7798915
  • definition thmcounterdefinition: Differential Privacy NEURIPS2020_fc4ddc15
  • theorem thmcountertheorem: Linear Convergence Theorem