Near-Optimal Reinforcement Learning with Shuffle Differential Privacy

TL;DR

This work tackles privacy in online episodic reinforcement learning for networked systems by adopting the shuffle differential privacy (SDP) model, which balances privacy with utility without a trusted central server. It introduces SDP-PE, a policy-elimination RL algorithm designed for the SDP setting, featuring an exponential batch scheduling and a forgetting mechanism to manage exploration, privacy, and switching costs. Theoretical results show a near-optimal regret bound of $\tilde{O}(\sqrt{XAH^3K} + X^3AH^6/\varepsilon)$ with policy switches $O(H\log K)$, supported by empirical results on RiverSwim and synthetic MAB benchmarks that confirm favorable privacy-utility trade-offs and dramatic reductions in deployment costs. Overall, the paper demonstrates the viability of the shuffle privacy model for secure, data-driven decision-making in large-scale networked RL settings, offering a practical privacy-respecting alternative to fully centralized or local privacy approaches.

Abstract

Reinforcement learning (RL) is a powerful tool for sequential decision-making, but its application is often hindered by privacy concerns arising from its interaction data. This challenge is particularly acute in advanced networked systems, where learning from operational and user data can expose systems to privacy inference attacks. Existing differential privacy (DP) models for RL are often inadequate: the centralized model requires a fully trusted server, creating a single point of failure risk, while the local model incurs significant performance degradation that is unsuitable for many networked applications. This paper addresses this gap by leveraging the emerging shuffle model of privacy, an intermediate trust model that provides strong privacy guarantees without a centralized trust assumption. We present Shuffle Differentially Private Policy Elimination (SDP-PE), the first generic policy elimination-based algorithm for episodic RL under the shuffle model. Our method introduces a novel exponential batching schedule and a ``forgetting'' mechanism to balance the competing demands of privacy and learning performance. Our analysis shows that SDP-PE achieves a near-optimal regret bound, demonstrating a superior privacy-regret trade-off with utility comparable to the centralized model while significantly outperforming the local model. The numerical experiments also corroborate our theoretical results and demonstrate the effectiveness of SDP-PE. This work establishes the viability of the shuffle model for secure data-driven decision-making in networked systems.

Figures (8)

• Figure 1: An illustration of online reinforcement learning in a distributed network. A central coordinator performs "centralized learning", and provides service for online streaming user requests in distributed nodes by policy deployment. Each local node generates episodic interaction data. This sensitive interaction data is then (privately) transmitted back to the coordinator for global policy updates.
• Figure 2: Illustration of the operational workflows in the centralized, local, and shuffle privacy models.
• Figure 3: Overview of the SDP-PE algorithm (Algorithm \ref{['algo: Private Batch-based Policy Elimination']}). The central coordinator deploys a global policy to distributed nodes, which process batched user requests. Private interaction data is then shuffled and transmitted back for policy value updates and elimination.
• Figure 4: Example of policy elimination. The LCB of the estimated optimal policy $\pi_2$ is $0.75$. Policies $\pi_3$ and $\pi_5$, whose UCB are below this threshold, are identified as sub-optimal and consequently eliminated.
• Figure 5: Simplified RiverSwim MDP -- solid and dotted arrows denote the transitions under actions "right" and "left", respectively.

Theorems & Definitions (20)

• Definition 1: Differential Privacy (DP)
• Definition 2: Shuffle Differential Privacy (SDP)
• Definition 3: $B$-batch SDP
• Definition 5: Abosorbing MDP $P'$
• Theorem 6: Performance guarantee of SDP-PE
• Lemma 7
• Lemma 8
• Lemma 9
• Lemma 10: Shuffle DP Privatizer
• Corollary 11: Regret Bound under SDP