Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Mastering the Game of Go with Self-play Experience Replay

Jingbin Liu, Xuechun Wang

TL;DR

This work introduces QZero, a model-free off-policy reinforcement learning algorithm that learns a Nash-equilibrium policy for Go using self-play and a single Q-value network, avoiding environment models and search at training time. By incorporating entropy regularization, an ignition mechanism, and Polyak-averaged targets, QZero trains with off-policy experience replay and achieves Go-level performance comparable to AlphaGo under modest hardware (7 GPUs). The experiments demonstrate Go mastery without human data or MCTS during training, highlighting the feasibility and efficiency of large-scale off-policy RL for complex domains. The findings suggest that learning-based approaches can rival planning-based systems in rich environments and motivate further exploration of continued learning and scalability in RL.

Abstract

The game of Go has long served as a benchmark for artificial intelligence, demanding sophisticated strategic reasoning and long-term planning. Previous approaches such as AlphaGo and its successors, have predominantly relied on model-based Monte-Carlo Tree Search (MCTS). In this work, we present QZero, a novel model-free reinforcement learning algorithm that forgoes search during training and learns a Nash equilibrium policy through self-play and off-policy experience replay. Built upon entropy-regularized Q-learning, QZero utilizes a single Q-value network to unify policy evaluation and improvement. Starting tabula rasa without human data and trained for 5 months with modest compute resources (7 GPUs), QZero achieved a performance level comparable to that of AlphaGo. This demonstrates, for the first time, the efficiency of using model-free reinforcement learning to master the game of Go, as well as the feasibility of off-policy reinforcement learning in solving large-scale and complex environments.

Mastering the Game of Go with Self-play Experience Replay

TL;DR

This work introduces QZero, a model-free off-policy reinforcement learning algorithm that learns a Nash-equilibrium policy for Go using self-play and a single Q-value network, avoiding environment models and search at training time. By incorporating entropy regularization, an ignition mechanism, and Polyak-averaged targets, QZero trains with off-policy experience replay and achieves Go-level performance comparable to AlphaGo under modest hardware (7 GPUs). The experiments demonstrate Go mastery without human data or MCTS during training, highlighting the feasibility and efficiency of large-scale off-policy RL for complex domains. The findings suggest that learning-based approaches can rival planning-based systems in rich environments and motivate further exploration of continued learning and scalability in RL.

Abstract

The game of Go has long served as a benchmark for artificial intelligence, demanding sophisticated strategic reasoning and long-term planning. Previous approaches such as AlphaGo and its successors, have predominantly relied on model-based Monte-Carlo Tree Search (MCTS). In this work, we present QZero, a novel model-free reinforcement learning algorithm that forgoes search during training and learns a Nash equilibrium policy through self-play and off-policy experience replay. Built upon entropy-regularized Q-learning, QZero utilizes a single Q-value network to unify policy evaluation and improvement. Starting tabula rasa without human data and trained for 5 months with modest compute resources (7 GPUs), QZero achieved a performance level comparable to that of AlphaGo. This demonstrates, for the first time, the efficiency of using model-free reinforcement learning to master the game of Go, as well as the feasibility of off-policy reinforcement learning in solving large-scale and complex environments.
Paper Structure (13 sections, 8 equations, 5 figures, 2 tables, 1 algorithm)

This paper contains 13 sections, 8 equations, 5 figures, 2 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Methods
  3. Model-free reinforcement learning
  4. Unification under entropy regularization
  5. QZero
  6. Experiments
  7. Go environment
  8. Neural network architecture
  9. Infrastructure
  10. Training details
  11. Results
  12. Discussion
  13. Conclusion

Figures (5)

  • Figure 1: Neural network architectures of AlphaGo and QZero. a. AlphaGo uses two networks to represent policy and value, respectively. The two networks can be configured to either share network backbone parameters or be independent. b. QZero uses only one network to represent Q-value. Policy and value can be derived from the single network.
  • Figure 2: Training framework of QZero. Actors, Replay Buffer, and Learner constitute the training loop. Evaluators are attached to the training loop for monitoring the training process.
  • Figure 3: Evaluations. a. The history score gain curve of QZero training. b. Elo ratings of raw neural networks of QZero, AlphaGo and AlphaGo Zero.
  • Figure 4: Action probability maps by the raw neural network of QZero. a. The probability map for the first move perfectly preserves the symmetries of the Go board. b. In the early stage of the game, the corners have higher value. c. As the game progresses, the AI gradually advances into the central region.
  • Figure 5: Self-play games of the raw neural network of QZero. a. Game generated using argmax with the neural network parameters from the 3rd month. b. Game generated using argmax with the neural network parameters from the 5th month. c. Game generated using probabilistic sampling.