Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Trustworthiness of Stochastic Gradient Descent in Distributed Learning

Hongyang Li, Caesar Wu, Mohammed Chadli, Said Mammar, Pascal Bouvry

TL;DR

The paper addresses the privacy risk of gradients in distributed learning under compressed SGD. It combines theoretical convergence insights for PowerSGD and Top-K SGD with empirical evaluations of GradInv and MIA across diverse datasets, showing that compression can decrease leakage in gradient inversion attacks, while MIA performance remains largely unaffected. The key finding is that uncompressed SGD poses higher privacy risk than compressed variants, though MIA may not be a reliable privacy metric in this context. The work underscores the need for improved privacy evaluation methods and motivates studying additional attack models to holistically assess the trustworthiness of gradient-compression techniques in distributed learning.

Abstract

Distributed learning (DL) uses multiple nodes to accelerate training, enabling efficient optimization of large-scale models. Stochastic Gradient Descent (SGD), a key optimization algorithm, plays a central role in this process. However, communication bottlenecks often limit scalability and efficiency, leading to increasing adoption of compressed SGD techniques to alleviate these challenges. Despite addressing communication overheads, compressed SGD introduces trustworthiness concerns, as gradient exchanges among nodes are vulnerable to attacks like gradient inversion (GradInv) and membership inference attacks (MIA). The trustworthiness of compressed SGD remains unexplored, leaving important questions about its reliability unanswered. In this paper, we provide a trustworthiness evaluation of compressed versus uncompressed SGD. Specifically, we conducted empirical studies using GradInv attacks, revealing that compressed SGD demonstrates significantly higher resistance to privacy leakage compared to uncompressed SGD. In addition, our findings suggest that MIA may not be a reliable metric for assessing privacy risks in distributed learning.

Trustworthiness of Stochastic Gradient Descent in Distributed Learning

TL;DR

The paper addresses the privacy risk of gradients in distributed learning under compressed SGD. It combines theoretical convergence insights for PowerSGD and Top-K SGD with empirical evaluations of GradInv and MIA across diverse datasets, showing that compression can decrease leakage in gradient inversion attacks, while MIA performance remains largely unaffected. The key finding is that uncompressed SGD poses higher privacy risk than compressed variants, though MIA may not be a reliable privacy metric in this context. The work underscores the need for improved privacy evaluation methods and motivates studying additional attack models to holistically assess the trustworthiness of gradient-compression techniques in distributed learning.

Abstract

Distributed learning (DL) uses multiple nodes to accelerate training, enabling efficient optimization of large-scale models. Stochastic Gradient Descent (SGD), a key optimization algorithm, plays a central role in this process. However, communication bottlenecks often limit scalability and efficiency, leading to increasing adoption of compressed SGD techniques to alleviate these challenges. Despite addressing communication overheads, compressed SGD introduces trustworthiness concerns, as gradient exchanges among nodes are vulnerable to attacks like gradient inversion (GradInv) and membership inference attacks (MIA). The trustworthiness of compressed SGD remains unexplored, leaving important questions about its reliability unanswered. In this paper, we provide a trustworthiness evaluation of compressed versus uncompressed SGD. Specifically, we conducted empirical studies using GradInv attacks, revealing that compressed SGD demonstrates significantly higher resistance to privacy leakage compared to uncompressed SGD. In addition, our findings suggest that MIA may not be a reliable metric for assessing privacy risks in distributed learning.

Paper Structure

This paper contains 14 sections, 8 equations, 1 figure, 2 tables.

Table of Contents

  1. Introduction
  2. Related Work and Problem Setup
  3. Gradient Compression
  4. Top-K SGD and PowerSGD
  5. Passive Attackers: GradInv and MIA
  6. Distributed Learning Setting
  7. GradInv Description
  8. Theoretical Analysis
  9. Convergence Analysis
  10. Empirical Study
  11. Experiments Setup
  12. Results and Analysis
  13. Discussion
  14. Conclusion

Figures (1)

  • Figure 1: Gradient Inversion Images Across Datesets