Privacy-Constrained Policies via Mutual Information Regularized Policy Gradients

TL;DR

The paper tackles privacy in reinforcement learning by enforcing a mutual information constraint between sensitive state variables and the agent's actions. It introduces a MI-based regularizer $I_{q_phi}(a_t;u_t)$ and develops three gradient-estimation strategies (model-based, model-free upper bound, and differentiable-simulator reparameterization) to optimize privacy-constrained policies. A dual formulation with Lagrange multipliers guides constrained optimization, and the authors explore multiple threat models, demonstrating that policies can hinder information leakage while maintaining strong task performance across tabular, continuous, and differentiable-robotics environments. These results suggest practical routes to increasing user trust in RL systems by controlling information disclosure without sacrificing reward.

Abstract

As reinforcement learning techniques are increasingly applied to real-world decision problems, attention has turned to how these algorithms use potentially sensitive information. We consider the task of training a policy that maximizes reward while minimizing disclosure of certain sensitive state variables through the actions. We give examples of how this setting covers real-world problems in privacy for sequential decision-making. We solve this problem in the policy gradients framework by introducing a regularizer based on the mutual information (MI) between the sensitive state and the actions. We develop a model-based stochastic gradient estimator for optimization of privacy-constrained policies. We also discuss an alternative MI regularizer that serves as an upper bound to our main MI regularizer and can be optimized in a model-free setting, and a powerful direct estimator that can be used in an environment with differentiable dynamics. We contrast previous work in differentially-private RL to our mutual-information formulation of information disclosure. Experimental results show that our training method results in policies that hide the sensitive state, even in challenging high-dimensional tasks.

Figures (11)

• Figure 1: Action distribution in four states in the VPN MDP with four mirrors. Top to bottom: our approach with $\boldsymbol{\lambda}=0$, $\boldsymbol{\lambda}=1$, DPQL with $\sigma=0.1$, $\sigma=5.0$.
• Figure 2: Trajectories for the 2d control task, $u$-unconstrained (left) and -constrained (right). The policy induces more variance in the $u$-direction in the constrained case, with less mutual information between $a_t$ and $u_t$. Our policy reduces the MI to zero when computed with a Gaussian discriminator, but this diverges from the MI as estimated by a nonparametric KDE at later timesteps as $u$ is less Gaussian.
• Figure 3: Simulated robotics policies trained with mutual information constraints. In multi-pusher (upper row), the goal is red; movable balls are cyan and orange. The unconstrained policy moves the active ball to the goal, while the constrained policy moves both balls to the goal. For turning-ant (lower row), the private policy moves diagonally while the unconstrained policy moves exactly in the direction of highest reward.
• Figure 4: Left: Multi-Pusher, Right: Turning-ant. Full-trajectory MI and reward for different levels of truncated MI constraint. In both MDPs, we find policies that reduce the disclosure of sensitive variables. In multi-pusher, we can reduce disclosure significantly with a minor drop in reward. In turning-ant, there is a trade-off between disclosure and reward.
• Figure 5: Trajectories from the internet connectivity environment. One the left we have a Lagrange multiplier $\boldsymbol{\lambda} = \boldsymbol{0}$, while on the right we have $\boldsymbol{\lambda} = \boldsymbol{1}$. We see that the trajectories in the constrained case are able to completely remove the mutual information between the action and the sensitive state by choosing a policy of always activating the VPN and then choosing mirror 0.

Theorems & Definitions (6)

• Theorem 2.1
• proof
• Lemma A.1
• proof
• Lemma A.2
• proof