Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Beyond Average Return in Markov Decision Processes

Alexandre Marthe, Aurélien Garivier, Claire Vernade

TL;DR

It is proved that only generalized means can be optimized exactly, even in the more general framework of Distributional Reinforcement Learning (DistRL), and provided error bounds on the resulting estimators.

Abstract

What are the functionals of the reward that can be computed and optimized exactly in Markov Decision Processes?In the finite-horizon, undiscounted setting, Dynamic Programming (DP) can only handle these operations efficiently for certain classes of statistics. We summarize the characterization of these classes for policy evaluation, and give a new answer for the planning problem. Interestingly, we prove that only generalized means can be optimized exactly, even in the more general framework of Distributional Reinforcement Learning (DistRL).DistRL permits, however, to evaluate other functionals approximately. We provide error bounds on the resulting estimators, and discuss the potential of this approach as well as its limitations.These results contribute to advancing the theory of Markov Decision Processes by examining overall characteristics of the return, and particularly risk-conscious strategies.

Beyond Average Return in Markov Decision Processes

TL;DR

It is proved that only generalized means can be optimized exactly, even in the more general framework of Distributional Reinforcement Learning (DistRL), and provided error bounds on the resulting estimators.

Abstract

What are the functionals of the reward that can be computed and optimized exactly in Markov Decision Processes?In the finite-horizon, undiscounted setting, Dynamic Programming (DP) can only handle these operations efficiently for certain classes of statistics. We summarize the characterization of these classes for policy evaluation, and give a new answer for the planning problem. Interestingly, we prove that only generalized means can be optimized exactly, even in the more general framework of Distributional Reinforcement Learning (DistRL).DistRL permits, however, to evaluate other functionals approximately. We provide error bounds on the resulting estimators, and discuss the potential of this approach as well as its limitations.These results contribute to advancing the theory of Markov Decision Processes by examining overall characteristics of the return, and particularly risk-conscious strategies.
Paper Structure (29 sections, 8 theorems, 40 equations, 4 figures, 3 algorithms)

This paper contains 29 sections, 8 theorems, 40 equations, 4 figures, 3 algorithms.

Table of Contents

  1. Introduction
  2. Background
  3. Distributional RL
  4. Beyond expected returns
  5. Policy Evaluation
  6. Experiment: empirical validation of the bounds on a simple MDP
  7. Planning
  8. Q-Learning Exponential Utilities
  9. Discussions and Related Work
  10. The Discounted Framework
  11. Utilizing functionals to optimize expected return.
  12. Dynamic programming for the optimization of other functionals
  13. Conclusion
  14. Additional remarks
  15. Remarks on the recursive definition of the Q-value distribution
  16. ...and 14 more sections

Key Result

Proposition 1

Let $\pi$ be a policy and $\eta_{}^{\pi}$ the associated Q-value distributions. Assume the return is bounded on a interval of length $\Delta_\eta \leq H\Delta_R$, where $\Delta_R$ is the support size of the reward distribution. Let $\hat{\eta}_{}^{\pi}$ be the Q-value distributions obtained by dynam

Figures (4)

  • Figure 1: A Chain MDP of length $H$ with deterministic transition and identical reward distribution for each state.
  • Figure 2: Left: Validation of Theorem \ref{['th:eval_stat_func_error']} on CVaR$(\alpha)$ together with the scaled upper bound (see main text for discussion): the quadratic dependence in $H$ is verified. Right: Validation of Proposition \ref{['prop:error_dist']}: The cumulative projection error (dashed blue) is the sum of the projection errors at every time step, and matches the true approximation error (solid blue). The theoretical upper bound (dashed red) differs only by a factor 2.
  • Figure 3: Evaluation of the Wasserstein Distance between the true value distribution and the approximated one, in the MDP described in Corollary \ref{['cor:mdp_proj_tight']}
  • Figure 4: Left: Independence Property Counter Example, Right: Translation Property Counter Example. Each arrow represents a state transition, which is characterized by the action leading to the transition, the probability of such transition, and the reward distribution of the transition.

Theorems & Definitions (17)

  • Definition 1: Bellman closedness rowland_statistics_2019
  • Proposition 1
  • Lemma 1
  • Theorem 1
  • Definition 2: Bellman Optimizable statistical functional
  • Remark
  • Lemma 2
  • Theorem 2
  • proof
  • proof
  • ...and 7 more