Decision Making in Non-Stationary Environments with Policy-Augmented Search

TL;DR

This work tackles sequential decision-making in non-stationary environments by introducing Policy-Augmented Monte Carlo Tree Search (PA-MCTS), which selects actions via a convex combination of stale policy $Q$-values and current MCTS estimates: $\alpha Q_{0}^{\pi^{*}_{0}}(s, a) + (1-\alpha)\overline{G}_{t}(s, a)$. The authors provide theoretical bounds on how much $Q$-value drift can occur under bounded transition changes and prove conditions under which PA-MCTS yields the optimal one-step action and bounds the cumulative regret. Empirically, PA-MCTS outperforms baselines like AlphaZero and DDQN on four OpenAI Gym tasks under non-stationarity, offering faster convergence and robustness under time constraints. The work demonstrates that leveraging an updated environment model together with offline policy information can significantly improve decision-making in changing environments with limited planning time, with broad implications for real-time autonomous systems.

Abstract

Sequential decision-making under uncertainty is present in many important problems. Two popular approaches for tackling such problems are reinforcement learning and online search (e.g., Monte Carlo tree search). While the former learns a policy by interacting with the environment (typically done before execution), the latter uses a generative model of the environment to sample promising action trajectories at decision time. Decision-making is particularly challenging in non-stationary environments, where the environment in which an agent operates can change over time. Both approaches have shortcomings in such settings -- on the one hand, policies learned before execution become stale when the environment changes and relearning takes both time and computational effort. Online search, on the other hand, can return sub-optimal actions when there are limitations on allowed runtime. In this paper, we introduce \textit{Policy-Augmented Monte Carlo tree search} (PA-MCTS), which combines action-value estimates from an out-of-date policy with an online search using an up-to-date model of the environment. We prove theoretical results showing conditions under which PA-MCTS selects the one-step optimal action and also bound the error accrued while following PA-MCTS as a policy. We compare and contrast our approach with AlphaZero, another hybrid planning approach, and Deep Q Learning on several OpenAI Gym environments. Through extensive experiments, we show that under non-stationary settings with limited time constraints, PA-MCTS outperforms these baselines.

Figures (21)

• Figure 1: The Frozen Lake environment involves an agent trying to cross a frozen lake from a start position to a goal position (pre-determined) without falling into any holes
• Figure 2: We show the cumulative reward obtained by PA-MCTS, standard MCTS ($\alpha = 0$), and DQN ($\alpha = 1$) for different environmental changes. Note that as the environment changes, DQN achieves close to 0 rewards. PA-MCTS outperforms both baseline approaches.
• Figure 3: We show the optimal $\alpha$ for varying MCTS iterations and environmental conditions.
• Figure 4: The performance of PA-MCTS and AlphaZero under varying levels of noise with $g=500\,\text{m/s}^2$. PA-MCTS outperforms AlphaZero in most settings.
• Figure 5: We compare PA-MCTS with $\alpha$-selection with AlphaZero. While both approaches find the optimal policy, PA-MCTS converges significantly faster.

Theorems & Definitions (16)

• Theorem 1
• Theorem 2
• Corollary 2.1
• Corollary 2.2
• Corollary 2.3
• Proposition 1
• Proposition 2
• Theorem 3
• proof
• proof