Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Entropic Risk-Aware Monte Carlo Tree Search

Pedro P. Santos, Jacopo Silvestrin, Alberto Sardinha, Francisco S. Melo

TL;DR

This work introduces ERM-MCTS for solving risk-aware Markov decision processes under the entropic risk measure (ERM). By replacing mean estimators with ERM estimators and adopting depth-dependent risk parameters, the method achieves provable correctness and polynomial concentration, deriving non-asymptotic guarantees via non-stationary bandit analyses that are then extended recursively to trees. The authors develop UCB-style algorithms for both stationary and non-stationary, deterministic and non-deterministic bandits, establishing convergence of the root ERM to the optimal value $V^*_0(s_0)$ and polynomial tail bounds for estimation errors. Empirical results on a four-state MDP (MDP-4) show ERM-MCTS closely matches an oracle backward-induction baseline and outperforms a general MCTS variant, with clear sensitivity to the risk parameter $\beta$. Overall, the paper provides the first theoretically grounded ERM-based MCTS framework for risk-aware planning, with implications for robust sequential decision-making in large decision spaces.

Abstract

We propose a provably correct Monte Carlo tree search (MCTS) algorithm for solving \textit{risk-aware} Markov decision processes (MDPs) with \textit{entropic risk measure} (ERM) objectives. We provide a \textit{non-asymptotic} analysis of our proposed algorithm, showing that the algorithm: (i) is \textit{correct} in the sense that the empirical ERM obtained at the root node converges to the optimal ERM; and (ii) enjoys \textit{polynomial regret concentration}. Our algorithm successfully exploits the dynamic programming formulations for solving risk-aware MDPs with ERM objectives introduced by previous works in the context of an upper confidence bound-based tree search algorithm. Finally, we provide a set of illustrative experiments comparing our risk-aware MCTS method against relevant baselines.

Entropic Risk-Aware Monte Carlo Tree Search

TL;DR

This work introduces ERM-MCTS for solving risk-aware Markov decision processes under the entropic risk measure (ERM). By replacing mean estimators with ERM estimators and adopting depth-dependent risk parameters, the method achieves provable correctness and polynomial concentration, deriving non-asymptotic guarantees via non-stationary bandit analyses that are then extended recursively to trees. The authors develop UCB-style algorithms for both stationary and non-stationary, deterministic and non-deterministic bandits, establishing convergence of the root ERM to the optimal value and polynomial tail bounds for estimation errors. Empirical results on a four-state MDP (MDP-4) show ERM-MCTS closely matches an oracle backward-induction baseline and outperforms a general MCTS variant, with clear sensitivity to the risk parameter . Overall, the paper provides the first theoretically grounded ERM-based MCTS framework for risk-aware planning, with implications for robust sequential decision-making in large decision spaces.

Abstract

We propose a provably correct Monte Carlo tree search (MCTS) algorithm for solving \textit{risk-aware} Markov decision processes (MDPs) with \textit{entropic risk measure} (ERM) objectives. We provide a \textit{non-asymptotic} analysis of our proposed algorithm, showing that the algorithm: (i) is \textit{correct} in the sense that the empirical ERM obtained at the root node converges to the optimal ERM; and (ii) enjoys \textit{polynomial regret concentration}. Our algorithm successfully exploits the dynamic programming formulations for solving risk-aware MDPs with ERM objectives introduced by previous works in the context of an upper confidence bound-based tree search algorithm. Finally, we provide a set of illustrative experiments comparing our risk-aware MCTS method against relevant baselines.
Paper Structure (37 sections, 16 theorems, 142 equations, 7 figures, 2 tables)

This paper contains 37 sections, 16 theorems, 142 equations, 7 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Contributions
  3. Background
  4. Markov decision processes
  5. Risk-aware Markov decision processes
  6. Risk-aware multi-armed bandits
  7. Non-stationary risk-aware multi-armed bandits
  8. A UCB-based algorithm for the risk-aware non-stationary bandit
  9. Non-stationary non-deterministic risk-aware multi-armed bandits
  10. A UCB-based algorithm for the risk-aware non-stationary non-deterministic bandit
  11. Entropic risk-aware Monte Carlo tree search
  12. Illustrative experiments
  13. Conclusion
  14. Extended related work
  15. Risk-neutral MCTS
  16. ...and 22 more sections

Key Result

Theorem 1

Let $\bar{\rho}_n = \frac{1}{n} \sum_{i=1}^K T_i(n) \hat{\rho}_{i, T_i(n)}$ be the empirical $\text{ERM}_\beta$ when running algorithm eq:ucb_erm_algorithm under a non-stationary bandit satisfying Assumption assumption:non-stationary-bandit-assumption. Furthermore, for any $1/2 \le \eta < 1$, assume

Figures (7)

  • Figure 1: Illustration of the recursive relation between parameters $\{\alpha_h\}$, $\{\theta_h\}$, $\{\eta_h\}$, and $\{\xi_h\}$.
  • Figure 2: Risk-aware MCTS pseudocode.
  • Figure 3: MDP-4 environment illustration.
  • Figure 4: $\beta = 0.1$
  • Figure 5: $\beta = 0.5$
  • ...and 2 more figures

Theorems & Definitions (16)

  • Theorem 1
  • Theorem 2
  • Lemma 3
  • Theorem 4
  • Theorem 5
  • Theorem 6
  • Lemma 7: Properties of the ERM estimator
  • Lemma 8: Exponential concentration of the ERM estimator
  • Lemma 9: Polynomial concentration of the ERM estimator
  • Lemma 10
  • ...and 6 more