AUPO -- Abstracted Until Proven Otherwise: A Reward Distribution Based Abstraction Algorithm
Robin Schmöcker, Alexander Dockhorn, Bodo Rosenhahn
TL;DR
AUPO addresses the need for domain-independent, non-learning-based action abstraction in MCTS by exploiting reward distribution statistics collected during search to form root-action abstractions. It operates as a drop-in modification to the MCTS decision policy, requiring neither transition probabilities nor a DAG, and it can be combined with other tree-search abstractions. The authors provide theoretical guarantees of asymptotic soundness and finite-sample abstraction bounds, and empirical results on IPPC benchmarks show AUPO outperforming MCTS in the majority of tested environments. The work expands the applicability of MCTS in discrete, fully observable domains and offers a practical, low-overhead approach to improve planning performance via reward-distribution-based abstraction.
Abstract
We introduce a novel, drop-in modification to Monte Carlo Tree Search's (MCTS) decision policy that we call AUPO. Comparisons based on a range of IPPC benchmark problems show that AUPO clearly outperforms MCTS. AUPO is an automatic action abstraction algorithm that solely relies on reward distribution statistics acquired during the MCTS. Thus, unlike other automatic abstraction algorithms, AUPO requires neither access to transition probabilities nor does AUPO require a directed acyclic search graph to build its abstraction, allowing AUPO to detect symmetric actions that state-of-the-art frameworks like ASAP struggle with when the resulting symmetric states are far apart in state space. Furthermore, as AUPO only affects the decision policy, it is not mutually exclusive with other abstraction techniques that only affect the tree search.