Preserving the Privacy of Reward Functions in MDPs through Deception

TL;DR

The paper addresses the privacy of reward functions in MDP-based planning when decisions are observable and susceptible to IRL-based reward reconstruction. It shows that existing deception-based methods like MEIR leak reward information and proposes the Max Misinformation (MM) algorithm, which uses a carefully constructed anti-reward to simulate suboptimal trajectories while preserving a minimum expected reward $E_{min}$. MM is formulated as a (primal-dual) optimization, implementable via LP in discrete settings or primal-dual descent in continuous spaces, and enables generating anti-rewards through occupancy-measure or trajectory-distribution distances (with divergences such as $f$-divergences or IPMs). Empirically, MM outperforms DP-based privacy methods and prior deception approaches across Cybersecurity, Four Rooms, Frozen Lake, and Random MDP domains, and remains robust against observers who anticipate deception, thereby offering a practical approach to privacy-preserving reward learning in security-critical RL tasks.

Abstract

Preserving the privacy of preferences (or rewards) of a sequential decision-making agent when decisions are observable is crucial in many physical and cybersecurity domains. For instance, in wildlife monitoring, agents must allocate patrolling resources without revealing animal locations to poachers. This paper addresses privacy preservation in planning over a sequence of actions in MDPs, where the reward function represents the preference structure to be protected. Observers can use Inverse RL (IRL) to learn these preferences, making this a challenging task. Current research on differential privacy in reward functions fails to ensure guarantee on the minimum expected reward and offers theoretical guarantees that are inadequate against IRL-based observers. To bridge this gap, we propose a novel approach rooted in the theory of deception. Deception includes two models: dissimulation (hiding the truth) and simulation (showing the wrong). Our first contribution theoretically demonstrates significant privacy leaks in existing dissimulation-based methods. Our second contribution is a novel RL-based planning algorithm that uses simulation to effectively address these privacy concerns while ensuring a guarantee on the expected reward. Experiments on multiple benchmark problems show that our approach outperforms previous methods in preserving reward function privacy.

Figures (10)

• Figure 1: Flow of Information
• Figure 2: Expected Reward v/s Injected Noise of DQFN in the Four Rooms environment averaged over 5 seeds. The shaded region represents the max and min values. The large variance in the reward obtained underscores the difficulty in managing the privacy-reward tradeoff when using DQFN.
• Figure 3: Occupancy measures of different private policies satisfying the same reward constraint in the Four Rooms environment. The MM algorithm leads to policies that visit a diverse mix of high reward and low reward states.
• Figure 4: MM and MEIR against IQ-Learn given $10$ demonstrations. Figures correspond to (a) Cyber security domain, (b) Frozen Lake and (c) Random MDPs.
• Figure 5: MM and MEIR against MCE IRL with true occupancy measures. Figures correspond to (a) Four Rooms (b) Frozen Lake (c) Random MDPs (d) Cyber Security domain.

Theorems & Definitions (12)

• Lemma 1
• Theorem 1
• Lemma 2
• proof : Proof of Theorem \ref{['leak_theorem']}
• Proposition 1
• Lemma 1
• proof
• Proposition 1
• Lemma 2
• proof