Lipschitz Lifelong Monte Carlo Tree Search for Mastering Non-Stationary Tasks

TL;DR

Addresses non-stationary lifelong planning for MCTS by deriving an adaptive Upper Confidence Bound (aUCT) that fuses cross-task Lipschitz distance with sampling confidence. The LiZero framework integrates aUCT into MCTS decisions via $U_{\text{aUCT}}$ and combines it with standard UCT for robust search, accompanied by three distance-estimation approaches: data-driven, non-stationary policy-aware, and model-based neural-network distance, with $d(\mathcal{M},\mathcal{M}')\le(1+\kappa)L\hat{d}_{\text{para}}$ when neural models satisfy a Lipschitz condition. Theoretical results establish a positive acceleration factor $\Gamma>1$ and quantified sampling-efficiency gains, while practical online estimation guarantees support deployment. Empirical evaluation on a non-stationary grid-world suite shows LiZero achieves $3\sim4\times$ speedups and around $31\%$ higher early rewards, demonstrating substantial gains in dynamic decision-making tasks that evolve over time.

Abstract

Monte Carlo Tree Search (MCTS) has proven highly effective in solving complex planning tasks by balancing exploration and exploitation using Upper Confidence Bound for Trees (UCT). However, existing work have not considered MCTS-based lifelong planning, where an agent faces a non-stationary series of tasks -- e.g., with varying transition probabilities and rewards -- that are drawn sequentially throughout the operational lifetime. This paper presents LiZero for Lipschitz lifelong planning using MCTS. We propose a novel concept of adaptive UCT (aUCT) to transfer knowledge from a source task to the exploration/exploitation of a new task, depending on both the Lipschitz continuity between tasks and the confidence of knowledge in in Monte Carlo action sampling. We analyze LiZero's acceleration factor in terms of improved sampling efficiency and also develop efficient algorithms to compute aUCT in an online fashion by both data-driven and model-based approaches, whose sampling complexity and error bounds are also characterized. Experiment results show that LiZero significantly outperforms existing MCTS and lifelong learning baselines in terms of much faster convergence (3$\sim$4x) to optimal rewards. Our results highlight the potential of LiZero to advance decision-making and planning in dynamic real-world environments.

Figures (2)

• Figure 1: Comparing LiZero with MCTS and lifelong RL baselines. We demonstrate the convergence of different algorithms on representatives Tasks 1, 2, 6, and 10, in a non-stationary sequence of ten tasks. In Task 1, since no prior knowledge is yet available, our LiZero and other MCTS baselines show similar convergence speed and optimal rewards. From Task 2 to Task 10, as more knowledge from past tasks gets transferred to the new task by LiZero, it outperforms all baselines with more significantly improved convergence speed. In Task 10 with maximum past knowledge, LiZero demonstrates the largest improvement in convergence speed and optimal reward.
• Figure 2: In Figure \ref{['fig:per']}, LiZero shows a comfortable speedup of 3$\sim$4x, compared with MCTS and lifelong RL baselines, in terms of achieving the same level of optimal rewards with higher sample efficiency. In Figure \ref{['fig:ab']}, Our ablation study comparing different distance estimators in LiZero-U, LiZero-P, and LiZero-N, while MCTS-R can be viewed as a baseline without distance estimator. The relevant performance of these algorithms are provided in Table \ref{['tab:task']} and Figure \ref{['fig:per']} and thus not repeated here.The superior performance of LiZero is indeed resulted from the use of aUCT in MCTS. The tighter aUCT bounds we use, the higher performance we can obtain.

Theorems & Definitions (15)

• Definition 3.1
• Theorem 3.2: Lipschitz aUCT Rule
• Corollary 3.3: aUCT bound in lifelong planning
• Theorem 3.4
• Theorem 4.1: Sampling Complexity under Stationarity
• Theorem 4.2: Sampling Complexity under Non-Stationarity
• Theorem 4.3
• proof
• Definition A.2
• proof