Provable Partially Observable Reinforcement Learning with Privileged Information
Yang Cai, Xiangyu Liu, Argyris Oikonomou, Kaiqing Zhang
TL;DR
This work addresses the challenge of partial observability in reinforcement learning by systematically studying the use of privileged information during training. It analyzes two practical paradigms—expert policy distillation and asymmetric actor-critic—and identifies conditions under which these approaches become provably efficient, notably the deterministic filter condition and belief-learning mechanisms. The authors develop a theory-anchored suite of algorithms, including belief-weighted AAC and decoding-function learners, that achieve polynomial sample complexity and (quasi-)polynomial computational time, and extend these ideas to partially observable multi-agent settings under CTDE. The results illuminate when privileged information yields real gains in sample efficiency and computation, and offer a foundation for further advancing principled RL in structured POMDPs and MARL with privileged data.
Abstract
Partial observability of the underlying states generally presents significant challenges for reinforcement learning (RL). In practice, certain \emph{privileged information}, e.g., the access to states from simulators, has been exploited in training and has achieved prominent empirical successes. To better understand the benefits of privileged information, we revisit and examine several simple and practically used paradigms in this setting. Specifically, we first formalize the empirical paradigm of \emph{expert distillation} (also known as \emph{teacher-student} learning), demonstrating its pitfall in finding near-optimal policies. We then identify a condition of the partially observable environment, the \emph{deterministic filter condition}, under which expert distillation achieves sample and computational complexities that are \emph{both} polynomial. Furthermore, we investigate another useful empirical paradigm of \emph{asymmetric actor-critic}, and focus on the more challenging setting of observable partially observable Markov decision processes. We develop a belief-weighted asymmetric actor-critic algorithm with polynomial sample and quasi-polynomial computational complexities, in which one key component is a new provable oracle for learning belief states that preserve \emph{filter stability} under a misspecified model, which may be of independent interest. Finally, we also investigate the provable efficiency of partially observable multi-agent RL (MARL) with privileged information. We develop algorithms featuring \emph{centralized-training-with-decentralized-execution}, a popular framework in empirical MARL, with polynomial sample and (quasi-)polynomial computational complexities in both paradigms above. Compared with a few recent related theoretical studies, our focus is on understanding practically inspired algorithmic paradigms, without computationally intractable oracles.