CENSOR: Defense Against Gradient Inversion via Orthogonal Subspace Bayesian Sampling

TL;DR

Censor addresses gradient inversion in federated learning by restricting gradient updates to an orthogonal subspace to the original gradients and applying cold posterior Bayesian sampling to select the most favorable update. By projecting perturbations into a high-dimensional orthogonal space and guiding selection with loss reduction, it dramatically reduces the information leaked through gradients while maintaining model performance. Theoretical justification and extensive experiments across ImageNet, FFHQ, and CIFAR-10 show Censor outperforms state-of-the-art defenses against both stochastic-optimization and GAN-based inversions, including adaptive EOT attacks, with minimal computational overhead and no convergence degradation. This yields a practical privacy-preserving FL mechanism suitable for large neural networks.

Abstract

Federated learning collaboratively trains a neural network on a global server, where each local client receives the current global model weights and sends back parameter updates (gradients) based on its local private data. The process of sending these model updates may leak client's private data information. Existing gradient inversion attacks can exploit this vulnerability to recover private training instances from a client's gradient vectors. Recently, researchers have proposed advanced gradient inversion techniques that existing defenses struggle to handle effectively. In this work, we present a novel defense tailored for large neural network models. Our defense capitalizes on the high dimensionality of the model parameters to perturb gradients within a subspace orthogonal to the original gradient. By leveraging cold posteriors over orthogonal subspaces, our defense implements a refined gradient update mechanism. This enables the selection of an optimal gradient that not only safeguards against gradient inversion attacks but also maintains model utility. We conduct comprehensive experiments across three different datasets and evaluate our defense against various state-of-the-art attacks and defenses. Code is available at https://censor-gradient.github.io.

Figures (11)

• Figure 1: Intuition of cold Bayesian posteriors sampling over orthogonal subspace. The left gray plane denotes the subspace that resides original gradients $G^0$. The bottom blue plane signifies a gradient subspace orthogonal to the original gradient subspace (gray plane), denoted as $G^*$. In the middle of the figure, a 3D training loss landscape is projected onto the 2D plane, parallel to the orthogonal gradient subspace. Cold Bayesian posteriors sampling enables Censor to sample multiple gradients and select the optimal one. This is illustrated by three potential directions in the orthogonal subspace, with the red indicating the optimal gradient that strictly points towards the optimal loss reduction direction. Notably, the optimal gradient highlighted in red not only lies within the orthogonal subspace, effectively protecting data privacy, but also minimizes the training loss, thereby ensuring the model utility.
• Figure 2: Results of SOTA attacks inversions (batch size equals to 1) across the initial five epochs.
• Figure 3: GGL consistently inverts similar images across distinct inputs.
• Figure 4: Overview of Censor.
• Figure 5: Orthogonal projection.

Theorems & Definitions (1)

• Definition 2.1: Differential Privacy