Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Act as You Learn: Adaptive Decision-Making in Non-Stationary Markov Decision Processes

Baiting Luo, Yunuo Zhang, Abhishek Dubey, Ayan Mukhopadhyay

TL;DR

The work tackles online decision-making in non-stationary MDPs with discrete, unknown updates to dynamics. It introduces ADA-MCTS, a framework combining risk-averse MCTS with dual-phase adaptive sampling and latent-parameter transfer to learn updated dynamics online and reduce pessimism in well-understood regions. By quantifying epistemic and aleatoric uncertainty via Bayesian methods, ADA-MCTS switches from exploration-focused planning to reward-maximization as knowledge improves, while transferring useful prior knowledge through latent parameters and replay buffers. Empirical results on three benchmark environments show ADA-MCTS often outperforms state-of-the-art baselines in both performance and computation time, highlighting its practical value for safe, adaptive online planning in evolving environments.

Abstract

A fundamental (and largely open) challenge in sequential decision-making is dealing with non-stationary environments, where exogenous environmental conditions change over time. Such problems are traditionally modeled as non-stationary Markov decision processes (NSMDP). However, existing approaches for decision-making in NSMDPs have two major shortcomings: first, they assume that the updated environmental dynamics at the current time are known (although future dynamics can change); and second, planning is largely pessimistic, i.e., the agent acts ``safely'' to account for the non-stationary evolution of the environment. We argue that both these assumptions are invalid in practice -- updated environmental conditions are rarely known, and as the agent interacts with the environment, it can learn about the updated dynamics and avoid being pessimistic, at least in states whose dynamics it is confident about. We present a heuristic search algorithm called \textit{Adaptive Monte Carlo Tree Search (ADA-MCTS)} that addresses these challenges. We show that the agent can learn the updated dynamics of the environment over time and then act as it learns, i.e., if the agent is in a region of the state space about which it has updated knowledge, it can avoid being pessimistic. To quantify ``updated knowledge,'' we disintegrate the aleatoric and epistemic uncertainty in the agent's updated belief and show how the agent can use these estimates for decision-making. We compare the proposed approach with the multiple state-of-the-art approaches in decision-making across multiple well-established open-source problems and empirically show that our approach is faster and highly adaptive without sacrificing safety.

Act as You Learn: Adaptive Decision-Making in Non-Stationary Markov Decision Processes

TL;DR

The work tackles online decision-making in non-stationary MDPs with discrete, unknown updates to dynamics. It introduces ADA-MCTS, a framework combining risk-averse MCTS with dual-phase adaptive sampling and latent-parameter transfer to learn updated dynamics online and reduce pessimism in well-understood regions. By quantifying epistemic and aleatoric uncertainty via Bayesian methods, ADA-MCTS switches from exploration-focused planning to reward-maximization as knowledge improves, while transferring useful prior knowledge through latent parameters and replay buffers. Empirical results on three benchmark environments show ADA-MCTS often outperforms state-of-the-art baselines in both performance and computation time, highlighting its practical value for safe, adaptive online planning in evolving environments.

Abstract

A fundamental (and largely open) challenge in sequential decision-making is dealing with non-stationary environments, where exogenous environmental conditions change over time. Such problems are traditionally modeled as non-stationary Markov decision processes (NSMDP). However, existing approaches for decision-making in NSMDPs have two major shortcomings: first, they assume that the updated environmental dynamics at the current time are known (although future dynamics can change); and second, planning is largely pessimistic, i.e., the agent acts ``safely'' to account for the non-stationary evolution of the environment. We argue that both these assumptions are invalid in practice -- updated environmental conditions are rarely known, and as the agent interacts with the environment, it can learn about the updated dynamics and avoid being pessimistic, at least in states whose dynamics it is confident about. We present a heuristic search algorithm called \textit{Adaptive Monte Carlo Tree Search (ADA-MCTS)} that addresses these challenges. We show that the agent can learn the updated dynamics of the environment over time and then act as it learns, i.e., if the agent is in a region of the state space about which it has updated knowledge, it can avoid being pessimistic. To quantify ``updated knowledge,'' we disintegrate the aleatoric and epistemic uncertainty in the agent's updated belief and show how the agent can use these estimates for decision-making. We compare the proposed approach with the multiple state-of-the-art approaches in decision-making across multiple well-established open-source problems and empirically show that our approach is faster and highly adaptive without sacrificing safety.
Paper Structure (11 sections, 6 equations, 3 figures, 2 tables, 1 algorithm)

This paper contains 11 sections, 6 equations, 3 figures, 2 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Problem
  3. Non-Stationary Markov Decision Process (NS-MDP)
  4. Goals
  5. Approach
  6. Risk-Averse Monte Carlo Tree Search
  7. Dual Phase Adaptive Sampling
  8. Adaptive Monte Carlo Tree Search
  9. Experiments
  10. Related Work
  11. Conclusion

Figures (3)

  • Figure 1: Schematic representation of the proposed dual-phase adaptive sampling approach.
  • Figure 2: The three environments we use to validate our approach: (a) the frozen lake environment, where the agent must go from start to the goal without falling into the holes; (b) the cliff walking environment, where the agent must go from start to goal without falling off the cliff, but it must accrue a small penalty for each step it takes; and (c) the non-stationary bridge environment from DBLP:conf/nips/LecarpentierR19. We add an extra hole to make the environment more challenging. The agent must go from S to G through F (H denotes holes).
  • Figure 3: We conduct an ablation study to evaluate the importance of each component of our approach. The experiment highlights the need for effective knowledge transfer and risk-averse exploration.