Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

R-CONV: An Analytical Approach for Efficient Data Reconstruction via Convolutional Gradients

Tamer Ahmed Eltaras, Qutaibah Malluhi, Alessandro Savino, Stefano Di Carlo, Adnan Qayyum, Junaid Qadir

TL;DR

R-CONV presents an analytical framework for reconstructing training data from convolutional-layer gradients in federated learning, showing that gradient constraints can be more informative than weight constraints, especially as convolutional depth increases. By combining forward/backward equations and exploiting activation-derivative expressions tied to activation outputs, it reconstructs inputs even with non-invertible activations like ReLU. The approach leverages a rank-analysis of the combined forward/backward operators to characterize feasibility and demonstrates superior reconstruction quality and speed over prior analytic (R-GAP) and optimization-based (DLG) methods across multiple CNN architectures and datasets. Practically, this highlights a heightened privacy risk in gradient-sharing FL and motivates stronger defenses that consider gradient information, not just weight constraints.

Abstract

In the effort to learn from extensive collections of distributed data, federated learning has emerged as a promising approach for preserving privacy by using a gradient-sharing mechanism instead of exchanging raw data. However, recent studies show that private training data can be leaked through many gradient attacks. While previous analytical-based attacks have successfully reconstructed input data from fully connected layers, their effectiveness diminishes when applied to convolutional layers. This paper introduces an advanced data leakage method to efficiently exploit convolutional layers' gradients. We present a surprising finding: even with non-fully invertible activation functions, such as ReLU, we can analytically reconstruct training samples from the gradients. To the best of our knowledge, this is the first analytical approach that successfully reconstructs convolutional layer inputs directly from the gradients, bypassing the need to reconstruct layers' outputs. Prior research has mainly concentrated on the weight constraints of convolution layers, overlooking the significance of gradient constraints. Our findings demonstrate that existing analytical methods used to estimate the risk of gradient attacks lack accuracy. In some layers, attacks can be launched with less than 5% of the reported constraints.

R-CONV: An Analytical Approach for Efficient Data Reconstruction via Convolutional Gradients

TL;DR

R-CONV presents an analytical framework for reconstructing training data from convolutional-layer gradients in federated learning, showing that gradient constraints can be more informative than weight constraints, especially as convolutional depth increases. By combining forward/backward equations and exploiting activation-derivative expressions tied to activation outputs, it reconstructs inputs even with non-invertible activations like ReLU. The approach leverages a rank-analysis of the combined forward/backward operators to characterize feasibility and demonstrates superior reconstruction quality and speed over prior analytic (R-GAP) and optimization-based (DLG) methods across multiple CNN architectures and datasets. Practically, this highlights a heightened privacy risk in gradient-sharing FL and motivates stronger defenses that consider gradient information, not just weight constraints.

Abstract

In the effort to learn from extensive collections of distributed data, federated learning has emerged as a promising approach for preserving privacy by using a gradient-sharing mechanism instead of exchanging raw data. However, recent studies show that private training data can be leaked through many gradient attacks. While previous analytical-based attacks have successfully reconstructed input data from fully connected layers, their effectiveness diminishes when applied to convolutional layers. This paper introduces an advanced data leakage method to efficiently exploit convolutional layers' gradients. We present a surprising finding: even with non-fully invertible activation functions, such as ReLU, we can analytically reconstruct training samples from the gradients. To the best of our knowledge, this is the first analytical approach that successfully reconstructs convolutional layer inputs directly from the gradients, bypassing the need to reconstruct layers' outputs. Prior research has mainly concentrated on the weight constraints of convolution layers, overlooking the significance of gradient constraints. Our findings demonstrate that existing analytical methods used to estimate the risk of gradient attacks lack accuracy. In some layers, attacks can be launched with less than 5% of the reported constraints.
Paper Structure (11 sections, 33 equations, 3 figures, 2 tables)

This paper contains 11 sections, 33 equations, 3 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Methodology
  4. Gradient Computation and Input Reconstruction from Fully Connected Layer
  5. Propagating Gradient Through Activation Function
  6. Gradient Computation and Input Reconstruction in Convolution Layer
  7. Gradient Computation
  8. Input Reconstruction
  9. Analysis of Recursive Gradient on Convolutional Layers
  10. Results and Discussions
  11. Conclusions

Figures (3)

  • Figure 1: The four network architectures used for evaluation: (a) LeNet, as defined in DLG, serves as a baseline model. (b) CNN6, as defined in R-GAP, represents another standard architecture. (c) LeNet-O, a modified version of LeNet, demonstrates the effectiveness of our method in using fewer parameters for image reconstruction. (d) CNN6-O, a modified version of CNN6, reduces the number of filters in the last layer from 128 to 3, highlighting our method's ability to reconstruct images with minimal parameters. In the notation for each layer, such as "Conv1 5x5@12, /2", "5x5" specifies the kernel size, "12" represents the number of filters, and "/2" denotes the stride value.
  • Figure 2: Reconstruction quality by R-CONV and DLG. Subfigure (a) shows reconstruction results using images CIFAR-100, and (b) depicts reconstruction results using images from MNIST. The first row represents the original images, the second row shows reconstructions by R-CONV using LeNet-O, the third row shows reconstructions by DLG using LeNet, the fourth row shows reconstructions by R-CONV with ReLU activation, and the fifth row shows reconstructions by DLG with ReLU activation. The results indicate that R-CONV produces higher-quality reconstructions than DLG, and when using ReLU activation, DLG's reconstruction quality deteriorates significantly, whereas R-CONV maintains superior quality even with fewer parameters.
  • Figure 3: Reconstruction quality by R-CONV and R-GAP. This figure presents the reconstruction results of images from CIFAR-10, comparing R-CONV and R-GAP. The first row shows the original images, the second row displays reconstructions by R-CONV using CNN6-O, the third row shows reconstructions by R-GAP using CNN6, the fourth row presents reconstructions by R-CONV with ReLU activation, and the fifth row shows reconstructions by R-GAP with ReLU activation. The comparison underscores the superior performance of R-CONV, which maintains high-quality reconstructions even with ReLU activation, unlike R-GAP.