Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Gradient Inversion in Federated Reinforcement Learning

Shenghong He

TL;DR

The paper investigates privacy risks in Federated Reinforcement Learning by introducing RGIA, a gradient inversion attack augmented with prior-based regularizations on states, rewards, and transition dynamics to enforce environment plausibility. RGIA reduces the pseudo-solution problem and narrows the feasible reconstruction space, with theoretical guarantees and empirical validation across diverse control and driving tasks showing accurate recovery of local training data from shared gradients. The work demonstrates that naive gradient-based attacks can be mitigated by priors, but defenses such as differential privacy introduce trade-offs between privacy and policy performance, while HE and gradient quantization offer limited protection. Overall, RGIA exposes FRL privacy vulnerabilities, provides a framework for evaluating defenses, and motivates robust privacy-preserving mechanisms in distributed RL systems.

Abstract

Federated reinforcement learning (FRL) enables distributed learning of optimal policies while preserving local data privacy through gradient sharing.However, FRL faces the risk of data privacy leaks, where attackers exploit shared gradients to reconstruct local training data.Compared to traditional supervised federated learning, successful reconstruction in FRL requires the generated data not only to match the shared gradients but also to align with real transition dynamics of the environment (i.e., aligning with the real data transition distribution).To address this issue, we propose a novel attack method called Regularization Gradient Inversion Attack (RGIA), which enforces prior-knowledge-based regularization on states, rewards, and transition dynamics during the optimization process to ensure that the reconstructed data remain close to the true transition distribution.Theoretically, we prove that the prior-knowledge-based regularization term narrows the solution space from a broad set containing spurious solutions to a constrained subset that satisfies both gradient matching and true transition dynamics.Extensive experiments on control tasks and autonomous driving tasks demonstrate that RGIA can effectively constrain reconstructed data transition distributions and thus successfully reconstruct local private data.

Gradient Inversion in Federated Reinforcement Learning

TL;DR

The paper investigates privacy risks in Federated Reinforcement Learning by introducing RGIA, a gradient inversion attack augmented with prior-based regularizations on states, rewards, and transition dynamics to enforce environment plausibility. RGIA reduces the pseudo-solution problem and narrows the feasible reconstruction space, with theoretical guarantees and empirical validation across diverse control and driving tasks showing accurate recovery of local training data from shared gradients. The work demonstrates that naive gradient-based attacks can be mitigated by priors, but defenses such as differential privacy introduce trade-offs between privacy and policy performance, while HE and gradient quantization offer limited protection. Overall, RGIA exposes FRL privacy vulnerabilities, provides a framework for evaluating defenses, and motivates robust privacy-preserving mechanisms in distributed RL systems.

Abstract

Federated reinforcement learning (FRL) enables distributed learning of optimal policies while preserving local data privacy through gradient sharing.However, FRL faces the risk of data privacy leaks, where attackers exploit shared gradients to reconstruct local training data.Compared to traditional supervised federated learning, successful reconstruction in FRL requires the generated data not only to match the shared gradients but also to align with real transition dynamics of the environment (i.e., aligning with the real data transition distribution).To address this issue, we propose a novel attack method called Regularization Gradient Inversion Attack (RGIA), which enforces prior-knowledge-based regularization on states, rewards, and transition dynamics during the optimization process to ensure that the reconstructed data remain close to the true transition distribution.Theoretically, we prove that the prior-knowledge-based regularization term narrows the solution space from a broad set containing spurious solutions to a constrained subset that satisfies both gradient matching and true transition dynamics.Extensive experiments on control tasks and autonomous driving tasks demonstrate that RGIA can effectively constrain reconstructed data transition distributions and thus successfully reconstruct local private data.

Paper Structure

This paper contains 38 sections, 4 theorems, 33 equations, 9 figures, 10 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Gradient Inversion Attacks
  4. Adversarial Attacks in FRL
  5. Preliminary
  6. method
  7. Gradient Derivation Training Data
  8. Prior-knowledge-based Regularization Terms
  9. Regularization Analysis
  10. Experiments
  11. Experiment Setup
  12. Baseline Comparison Results
  13. Reconstructed State Distribution Analysis
  14. RGIA Defense Mechanism Analysis
  15. Leakage Gradient Scale Analysis
  16. ...and 23 more sections

Key Result

Lemma 1

Given the optimization problem: $\min_{\tilde{x}} J(\tilde{x}) = F(\tilde{x}) + \lambda \mathcal{R}_r(\tilde{r}), \quad \lambda > 0$, where $F(\tilde{x})$ is the primary objective function and $\mathcal{R}_r$ is the regularization term defined as: $\mathcal{R}_r(\tilde{r}) = \left(\text{ReLU}(\tilde

Figures (9)

  • Figure 1: The FRL learning framework. $\upsilon$ network is the value function network.
  • Figure 2: The reconstruction process of training data. The x-axis represents the number of iterations, and the y-axis corresponds to various evaluation metrics.
  • Figure 3: Visualization of the reconstruction process. Fake denotes the data reconstructed by gradient inversion attacks, while True refers to the ground-truth data.
  • Figure 4: The comparison results of low-dimensional environments. To ensure consistent visualization of coordinates, the state MSE is scaled up by $10^{-6}$ in the Hopper and Halfcheetah environments, and by $10^{-7}$ in Walker2d and Ant.
  • Figure 5: PCA visualization of reconstructed states from 10 random initializations.
  • ...and 4 more figures

Theorems & Definitions (11)

  • Definition 1: Sample and parameter space
  • Definition 2: Gradient inverse problem
  • Definition 3: Original solution space
  • Remark 1: State space constraints
  • Lemma 1: Reward space constraints
  • Remark 2
  • Theorem 1: Regularization for compressing the solution space
  • Lemma 2: Reward Space Constraint
  • proof
  • Theorem 2: Regularization for Compressing the Solution Space
  • ...and 1 more