Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

UVIP: Model-Free Approach to Evaluate Reinforcement Learning Algorithms

Denis Belomestny, Ilya Levin, Alexey Naumov, Sergey Samsonov

TL;DR

The paper addresses bounding the suboptimality gap $\Delta_\pi(x)=V^*(x)-V^\pi(x)$ in unknown environments by introducing UVIP, a model-free upper-value-iteration method that constructs almost-sure upper bounds $V^{\mathrm{up}}$ for policy and optimal values. UVIP leverages upper solutions to the Bellman optimality equation and martingale duality to produce an optimality certificate from samples without knowing the transition kernel. Theoretical results provide non-asymptotic convergence guarantees, variance bounds, and explicit error terms that depend on the suboptimality and sampling resources, while numerical results on discrete and continuous MDPs demonstrate tight bounds and competitive running times. Overall, UVIP offers a practical, principled tool for policy evaluation and confidence quantification in RL settings where the model is unknown and data are sample-based.

Abstract

Policy evaluation is an important instrument for the comparison of different algorithms in Reinforcement Learning (RL). However, even a precise knowledge of the value function $V^π$ corresponding to a policy $π$ does not provide reliable information on how far the policy $π$ is from the optimal one. We present a novel model-free upper value iteration procedure ({\sf UVIP}) that allows us to estimate the suboptimality gap $V^{\star}(x) - V^π(x)$ from above and to construct confidence intervals for \(V^\star\). Our approach relies on upper bounds to the solution of the Bellman optimality equation via the martingale approach. We provide theoretical guarantees for {\sf UVIP} under general assumptions and illustrate its performance on a number of benchmark RL problems.

UVIP: Model-Free Approach to Evaluate Reinforcement Learning Algorithms

TL;DR

The paper addresses bounding the suboptimality gap in unknown environments by introducing UVIP, a model-free upper-value-iteration method that constructs almost-sure upper bounds for policy and optimal values. UVIP leverages upper solutions to the Bellman optimality equation and martingale duality to produce an optimality certificate from samples without knowing the transition kernel. Theoretical results provide non-asymptotic convergence guarantees, variance bounds, and explicit error terms that depend on the suboptimality and sampling resources, while numerical results on discrete and continuous MDPs demonstrate tight bounds and competitive running times. Overall, UVIP offers a practical, principled tool for policy evaluation and confidence quantification in RL settings where the model is unknown and data are sample-based.

Abstract

Policy evaluation is an important instrument for the comparison of different algorithms in Reinforcement Learning (RL). However, even a precise knowledge of the value function corresponding to a policy does not provide reliable information on how far the policy is from the optimal one. We present a novel model-free upper value iteration procedure ({\sf UVIP}) that allows us to estimate the suboptimality gap from above and to construct confidence intervals for . Our approach relies on upper bounds to the solution of the Bellman optimality equation via the martingale approach. We provide theoretical guarantees for {\sf UVIP} under general assumptions and illustrate its performance on a number of benchmark RL problems.

Paper Structure

This paper contains 33 sections, 9 theorems, 93 equations, 4 figures, 2 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Contributions and Organization
  3. Notations and definitions.
  4. Preliminary
  5. Related Works
  6. Upper Solutions and the Main Concept of UVIP
  7. Comparison with PAC-based confidence intervals and IPOC framework.
  8. Approximate UVIP
  9. Theoretical Results
  10. Variance of the estimator and confidence bounds.
  11. Numerical Results
  12. Discrete state-space MDPs
  13. Continuous state-space MDPs
  14. Running time of the UVIP algorithm
  15. Conclusion and Future Work
  16. ...and 18 more sections

Key Result

Theorem 6.1

Let Aass:X -- Aass: r-Vpi lip hold and suppose that $\mathrm{Lip}_{\rho_{\mathsf{X}}}(\widehat{V}^{\rm{up}}_{0}) \leq L_0$ with some constant $L_0 > 0$. Then for any $k \in \mathbb{N}$ and $\delta \in (0,1)$, it holds with probability at least $1-\delta$ that In the above bound $\lesssim$ stands for inequality up to a constant depending on $\gamma, L_{\max}, L_{\psi}, L_{\pi}, L_{0}$ and $R_{\max

Figures (4)

  • Figure 1: The difference between $V^{\rm{up},\pi_i}(x)$ and $V^{\pi_i}(x)$. The x-axis represents states in a discrete environment for all pictures. Each group of three pictures of the same color illustrates the process of learning the policy from the first iteration to the last. First row: Evaluation of the policies during the process of Value Iteration for Garnet (left) and Chain environments (right). The policies are the greedy ones corresponding to the function $Q_i(x, a)$ at the $i$-th step. Second row: Comparison of the gap between $V^{\pi}$ and $V^{\rm{up}, \pi}$ for the learned policy $\pi$ and the corrupted policy $\pi_{c}$ in the NRoom environment. The color in these plots represents the value of $V^{\rm{up}, \pi} - V^{\pi}$.
  • Figure 2: Upper and lower bounds for three different policies. Left: For CartPole $\pi_1$, $\pi_2$, $\pi_3$ policies, respectively. For the horizontal axis, we sample a single trajectory according to the policy. Right: For Acrobot Dueling DQN and A2C policies, respectively. We evaluate the bounds for the first 50 states of the trajectory for each algorithm.
  • Figure 3: We illustrate the gap between $V^{\rm{up}, \pi}$ and $V^{\pi}$ in the TwinRooms environment. The color in these plots represents the value of $V^{\rm{up}, \pi} - V^{\pi}$. On the left and right, we show this quantity for $\pi_1$ and $\pi_2$, respectively. We obtain $\pi_1$ and $\pi_2$ after $2500$ and $5000$ learning steps of the Kernel-UCBVI algorithm.
  • Figure 4: Continuous grid-world environment with two rooms separated by a wall. The circles represent the regions with non-zero rewards.

Theorems & Definitions (22)

  • Definition 4.1
  • Remark 6.1
  • Theorem 6.1
  • proof
  • proof
  • Corollary 6.2
  • proof
  • Theorem 6.2
  • proof
  • Corollary 6.3
  • ...and 12 more