Efficient Policy Evaluation with Safety Constraint for Reinforcement Learning
Claire Chen, Shuze Daniel Liu, Shangtong Zhang
TL;DR
The paper tackles high variance in off-policy policy evaluation while enforcing execution safety in RL. It introduces a constrained variance-minimization framework that designs a behavior policy under safety thresholds, yielding unbiased PDIS estimates with reduced variance. Extending to sequential RL, it defines an extended reward tilde{r} and proves convex, feasible optimization problems that guarantee both variance reduction and safety; the SCOPE algorithm learns tilde{r} offline and computes the optimal safe policy mu^* via convex optimization. Empirically, SCOPE achieves substantial variance reduction at lower cost than baselines across Gridworld and MuJoCo, demonstrating the practicality of safe, offline-informed policy evaluation.
Abstract
In reinforcement learning, classic on-policy evaluation methods often suffer from high variance and require massive online data to attain the desired accuracy. Previous studies attempt to reduce evaluation variance by searching for or designing proper behavior policies to collect data. However, these approaches ignore the safety of such behavior policies -- the designed behavior policies have no safety guarantee and may lead to severe damage during online executions. In this paper, to address the challenge of reducing variance while ensuring safety simultaneously, we propose an optimal variance-minimizing behavior policy under safety constraints. Theoretically, while ensuring safety constraints, our evaluation method is unbiased and has lower variance than on-policy evaluation. Empirically, our method is the only existing method to achieve both substantial variance reduction and safety constraint satisfaction. Furthermore, we show our method is even superior to previous methods in both variance reduction and execution safety.