Towards Optimal Differentially Private Regret Bounds in Linear MDPs
Sharan Sahu
TL;DR
This paper addresses privacy-preserving regret minimization in episodic inhomogeneous linear MDPs under joint differential privacy, where transitions and rewards are linear in a feature map $\boldsymbol{\phi}(s,a)$. It introduces DP-LSVI-UCB++ by privatizing LSVI-UCB++ with Gaussian noise and GOE perturbations, and proves a regret bound of $\tilde{O}( d\sqrt{H^3K} + H^{15/4} d^{7/6} K^{1/2} / \epsilon )$ alongside $(\epsilon, \delta')$-JDP guarantees. The analysis leverages Bernstein-type martingale concentration and variance-aware weighting, enabling tighter regret bounds than previous private approaches. The work advances the privacy-utility trade-off for linear MDPs and motivates extensions to low-rank MDPs and adaptive noise schemes.
Abstract
We study regret minimization under privacy constraints in episodic inhomogeneous linear Markov Decision Processes (MDPs), motivated by the growing use of reinforcement learning (RL) in personalized decision-making systems that rely on sensitive user data. In this setting, both transition probabilities and reward functions are assumed to be linear in a feature mapping $φ(s, a)$, and we aim to ensure privacy through joint differential privacy (JDP), a relaxation of differential privacy suited to online learning. Prior work has established suboptimal regret bounds by privatizing the LSVI-UCB algorithm, which achieves $\widetilde{O}(\sqrt{d^3 H^4 K})$ regret in the non-private setting. Building on recent advances that improve this to near minimax optimal regret $\widetilde{O}(d\sqrt{H^{3}K})$ via LSVI-UCB++ with Bernstein-style bonuses, we design a new differentially private algorithm by privatizing LSVI-UCB++ and adapting techniques for variance-aware analysis from offline RL. Our algorithm achieves a regret bound of $\widetilde{O}(d \sqrt{H^3 K} + H^{15/4} d^{7/6} K^{1/2} / ε)$, improving over previous private methods. Empirical results show that our algorithm retains near-optimal utility compared to non-private baselines, indicating that privacy can be achieved with minimal performance degradation in this setting.