Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Revisiting Gradient Pruning: A Dual Realization for Defending against Gradient Attacks

Lulu Xue, Shengshan Hu, Ruizhi Zhao, Leo Yu Zhang, Shengqing Hu, Lichao Sun, Dezhong Yao

TL;DR

This work tackles gradient inversion attacks in collaborative learning by introducing Dual Gradient Pruning (DGP), a defense that simultaneously removes large and small gradient parameters to inflate reconstruction error while preserving model utility. Augmented with an error feedback mechanism, DGP achieves strong privacy guarantees and improved communication efficiency, with convergence guarantees similar to standard CL-SGD. Theoretical analysis shows formal privacy degradation for passive attackers and practical convergence bounds; extensive experiments demonstrate DGP's robustness against multiple GIAs and favorable accuracy and bandwidth metrics compared to existing defenses. A variant, ADGP, is proposed to further cut download costs by aligning gradient locations across users, though it relies on a trusted broadcaster; overall, the approach offers a practical, high-utility privacy-preserving alternative for collaborative learning and related domains.

Abstract

Collaborative learning (CL) is a distributed learning framework that aims to protect user privacy by allowing users to jointly train a model by sharing their gradient updates only. However, gradient inversion attacks (GIAs), which recover users' training data from shared gradients, impose severe privacy threats to CL. Existing defense methods adopt different techniques, e.g., differential privacy, cryptography, and perturbation defenses, to defend against the GIAs. Nevertheless, all current defense methods suffer from a poor trade-off between privacy, utility, and efficiency. To mitigate the weaknesses of existing solutions, we propose a novel defense method, Dual Gradient Pruning (DGP), based on gradient pruning, which can improve communication efficiency while preserving the utility and privacy of CL. Specifically, DGP slightly changes gradient pruning with a stronger privacy guarantee. And DGP can also significantly improve communication efficiency with a theoretical analysis of its convergence and generalization. Our extensive experiments show that DGP can effectively defend against the most powerful GIAs and reduce the communication cost without sacrificing the model's utility.

Revisiting Gradient Pruning: A Dual Realization for Defending against Gradient Attacks

TL;DR

This work tackles gradient inversion attacks in collaborative learning by introducing Dual Gradient Pruning (DGP), a defense that simultaneously removes large and small gradient parameters to inflate reconstruction error while preserving model utility. Augmented with an error feedback mechanism, DGP achieves strong privacy guarantees and improved communication efficiency, with convergence guarantees similar to standard CL-SGD. Theoretical analysis shows formal privacy degradation for passive attackers and practical convergence bounds; extensive experiments demonstrate DGP's robustness against multiple GIAs and favorable accuracy and bandwidth metrics compared to existing defenses. A variant, ADGP, is proposed to further cut download costs by aligning gradient locations across users, though it relies on a trusted broadcaster; overall, the approach offers a practical, high-utility privacy-preserving alternative for collaborative learning and related domains.

Abstract

Collaborative learning (CL) is a distributed learning framework that aims to protect user privacy by allowing users to jointly train a model by sharing their gradient updates only. However, gradient inversion attacks (GIAs), which recover users' training data from shared gradients, impose severe privacy threats to CL. Existing defense methods adopt different techniques, e.g., differential privacy, cryptography, and perturbation defenses, to defend against the GIAs. Nevertheless, all current defense methods suffer from a poor trade-off between privacy, utility, and efficiency. To mitigate the weaknesses of existing solutions, we propose a novel defense method, Dual Gradient Pruning (DGP), based on gradient pruning, which can improve communication efficiency while preserving the utility and privacy of CL. Specifically, DGP slightly changes gradient pruning with a stronger privacy guarantee. And DGP can also significantly improve communication efficiency with a theoretical analysis of its convergence and generalization. Our extensive experiments show that DGP can effectively defend against the most powerful GIAs and reduce the communication cost without sacrificing the model's utility.
Paper Structure (26 sections, 9 theorems, 21 equations, 10 figures, 7 tables, 2 algorithms)

This paper contains 26 sections, 9 theorems, 21 equations, 10 figures, 7 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Related Work
  3. Threat Model and Gradient Attacks
  4. Dual Gradient Pruning
  5. Analysis of Gradient Pruning
  6. Dual Gradient Pruning
  7. Theoretical Analysis
  8. Assumptions
  9. Security Analysis
  10. Convergence Guarantee
  11. Experiments
  12. Experimental Setup
  13. Privacy Evaluation
  14. Accuracy Evaluation
  15. Further Discussions
  16. ...and 11 more sections

Key Result

Proposition 1

For any given input $\mathbf{x}$ and shared model $\mathbf{W}$, the distance between the recovered data $\mathbf{x'}$ and the real data $\mathbf{x}$ is bounded by: where $\varphi$ is the mapping from input to the gradient, i.e., the reconstruction quality is limited by $||\varphi (\mathbf{x},\mathbf{W})- \varphi (\mathbf{x'},\mathbf{W})||_2=||\nabla\mathbf{W }- \mathbf{g} ||_2$.

Figures (10)

  • Figure 1: Relationship between relative gradient distance and reconstruction quality under IG (CIFAR10cifar with ResNet18 resnet18).
  • Figure 2: Comparison between Top-$k$ and DGP on privacy and accuracy ($20\%$ of parameters are selected in Top-$k$).
  • Figure 3: Data visualization on privacy evaluation by using multiple gradient inversion attacks.
  • Figure 4: Evaluation of model accuracy with different datasets and models (EF denotes the error feedback).
  • Figure 5: Reconstruction data visualization under GGL attack on ImageNet.
  • ...and 5 more figures

Theorems & Definitions (14)

  • Definition 1
  • Proposition 1
  • Theorem 1
  • Lemma 1
  • Theorem 2
  • Corollary 1
  • Proposition 2
  • proof
  • Theorem 3
  • proof
  • ...and 4 more