On the Sample Complexity of Differentially Private Policy Optimization
Yi He, Xingyu Zhou
TL;DR
The paper addresses the challenge of incorporating differential privacy into policy optimization and quantifies the resulting sample complexity costs. It introduces a PO-specific DP definition, a unifying batched one-pass meta-algorithm, and private variants DP-PG, DP-NPG, and DP-REBEL, analyzing their convergence under privacy. By reducing private PO to private regression subproblems, it leverages established privacy results to derive explicit, dimension-aware bounds, showing the privacy cost often appears as a lower-order term relative to the non-private rate. The results cover both first-order and global convergence, with specialized insights for Fisher-non-degenerate and tabular softmax settings, and provide practical guidance for privacy-preserving PO in sensitive applications like healthcare and RLHF. Experimental evidence in the appendix supports the theoretical findings and demonstrates privacy-utility trade-offs in a controlled setting.
Abstract
Policy optimization (PO) is a cornerstone of modern reinforcement learning (RL), with diverse applications spanning robotics, healthcare, and large language model training. The increasing deployment of PO in sensitive domains, however, raises significant privacy concerns. In this paper, we initiate a theoretical study of differentially private policy optimization, focusing explicitly on its sample complexity. We first formalize an appropriate definition of differential privacy (DP) tailored to PO, addressing the inherent challenges arising from on-policy learning dynamics and the subtlety involved in defining the unit of privacy. We then systematically analyze the sample complexity of widely-used PO algorithms, including policy gradient (PG), natural policy gradient (NPG) and more, under DP constraints and various settings, via a unified framework. Our theoretical results demonstrate that privacy costs can often manifest as lower-order terms in the sample complexity, while also highlighting subtle yet important observations in private PO settings. These offer valuable practical insights for privacy-preserving PO algorithms.