Simultaneous AlphaZero: Extending Tree Search to Markov Games

TL;DR

The paper tackles planning and learning in two-player zero-sum Markov games with simultaneous moves. It extends AlphaZero by treating decision nodes as matrix games and solving with a regret-optimal bandit solver within MCTS, using separate policy heads for each player. The authors provide theoretical guarantees on error propagation and exploitability, showing deeper search reduces root-value distortion. Empirical results on continuous Dubin Tag and space-domain awareness domains demonstrate robust, interpretable strategies that resist exploitation and improve with MCTS-guided search.

Abstract

Simultaneous AlphaZero extends the AlphaZero framework to multistep, two-player zero-sum deterministic Markov games with simultaneous actions. At each decision point, joint action selection is resolved via matrix games whose payoffs incorporate both immediate rewards and future value estimates. To handle uncertainty arising from bandit feedback during Monte Carlo Tree Search (MCTS), Simultaneous AlphaZero incorporates a regret-optimal solver for matrix games with bandit feedback. Simultaneous AlphaZero demonstrates robust strategies in a continuous-state discrete-action pursuit-evasion game and satellite custody maintenance scenarios, even when evaluated against maximally exploitative opponents.

Figures (8)

• Figure 1: Markov game tree representation with value function approximation
• Figure 2: Simultaneous AlphaZero network architecture for SDA game.
• Figure 3: Policy network best-response performance in the Dubin Tag environment.
• Figure 4: Exploitability of Simultaneous AlphaZero with increasing search iterations in Dubin Tag.
• Figure 5: Snapshots of the target in circular LEO and the corresponding observer value map $\hat{U}^1$. Higher intensity indicates higher value.

Theorems & Definitions (4)

• Lemma 1: Recursive error contraction
• proof
• Theorem 1: Root error from frontier error
• proof