Doubly Optimal Policy Evaluation for Reinforcement Learning
Shuze Daniel Liu, Claire Chen, Shangtong Zhang
TL;DR
This work tackles the high variance in off-policy policy evaluation by formulating a bi-level optimization that jointly optimizes data collection and data processing. The inner problem yields a closed-form optimal behavior policy $\u00mu^*$ proportional to $\pi_t(a|s)\sqrt{u_{pi,t}(s,a)}$, with $u_{pi,t}$ defined recursively via environment variance and future variance, while the outer problem yields an optimal baseline $b^*$ with $b_t^*(s,a)=q_{pi,t}(s,a)$. The result is a doubly optimal estimator that remains unbiased under an enlarged policy space $\u0004Lambda$ and achieves provably lower variance than on-policy Monte Carlo, offline data informed estimators, and doubly robust methods, with variance reductions compounding over the horizon. Empirically, the method (DOpt) substantially reduces variance and improves sample efficiency across Gridworld and MuJoCo, demonstrating practical impact for faster, cheaper policy evaluation in RL.
Abstract
Policy evaluation estimates the performance of a policy by (1) collecting data from the environment and (2) processing raw data into a meaningful estimate. Due to the sequential nature of reinforcement learning, any improper data-collecting policy or data-processing method substantially deteriorates the variance of evaluation results over long time steps. Thus, policy evaluation often suffers from large variance and requires massive data to achieve the desired accuracy. In this work, we design an optimal combination of data-collecting policy and data-processing baseline. Theoretically, we prove our doubly optimal policy evaluation method is unbiased and guaranteed to have lower variance than previously best-performing methods. Empirically, compared with previous works, we show our method reduces variance substantially and achieves superior empirical performance.