Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Superior Computer Chess with Model Predictive Control, Reinforcement Learning, and Rollout

Atharva Gundawar, Yuchao Li, Dimitri Bertsekas

TL;DR

A new architecture for move selection is introduced, within which available chess engines are used as components, and it is shown that this architecture improves substantially the performance of the position evaluation engine.

Abstract

In this paper we apply model predictive control (MPC), rollout, and reinforcement learning (RL) methodologies to computer chess. We introduce a new architecture for move selection, within which available chess engines are used as components. One engine is used to provide position evaluations in an approximation in value space MPC/RL scheme, while a second engine is used as nominal opponent, to emulate or approximate the moves of the true opponent player. We show that our architecture improves substantially the performance of the position evaluation engine. In other words our architecture provides an additional layer of intelligence, on top of the intelligence of the engines on which it is based. This is true for any engine, regardless of its strength: top engines such as Stockfish and Komodo Dragon (of varying strengths), as well as weaker engines. Structurally, our basic architecture selects moves by a one-move lookahead search, with an intermediate move generated by a nominal opponent engine, and followed by a position evaluation by another chess engine. Simpler schemes that forego the use of the nominal opponent, also perform better than the position evaluator, but not quite by as much. More complex schemes, involving multistep lookahead, may also be used and generally tend to perform better as the length of the lookahead increases. Theoretically, our methodology relies on generic cost improvement properties and the superlinear convergence framework of Newton's method, which fundamentally underlies approximation in value space, and related MPC/RL and rollout/policy iteration schemes. A critical requirement of this framework is that the first lookahead step should be executed exactly. This fact has guided our architectural choices, and is apparently an important factor in improving the performance of even the best available chess engines.

Superior Computer Chess with Model Predictive Control, Reinforcement Learning, and Rollout

TL;DR

A new architecture for move selection is introduced, within which available chess engines are used as components, and it is shown that this architecture improves substantially the performance of the position evaluation engine.

Abstract

In this paper we apply model predictive control (MPC), rollout, and reinforcement learning (RL) methodologies to computer chess. We introduce a new architecture for move selection, within which available chess engines are used as components. One engine is used to provide position evaluations in an approximation in value space MPC/RL scheme, while a second engine is used as nominal opponent, to emulate or approximate the moves of the true opponent player. We show that our architecture improves substantially the performance of the position evaluation engine. In other words our architecture provides an additional layer of intelligence, on top of the intelligence of the engines on which it is based. This is true for any engine, regardless of its strength: top engines such as Stockfish and Komodo Dragon (of varying strengths), as well as weaker engines. Structurally, our basic architecture selects moves by a one-move lookahead search, with an intermediate move generated by a nominal opponent engine, and followed by a position evaluation by another chess engine. Simpler schemes that forego the use of the nominal opponent, also perform better than the position evaluator, but not quite by as much. More complex schemes, involving multistep lookahead, may also be used and generally tend to perform better as the length of the lookahead increases. Theoretically, our methodology relies on generic cost improvement properties and the superlinear convergence framework of Newton's method, which fundamentally underlies approximation in value space, and related MPC/RL and rollout/policy iteration schemes. A critical requirement of this framework is that the first lookahead step should be executed exactly. This fact has guided our architectural choices, and is apparently an important factor in improving the performance of even the best available chess engines.
Paper Structure (8 sections, 13 equations, 3 figures, 4 tables)

This paper contains 8 sections, 13 equations, 3 figures, 4 tables.

Table of Contents

  1. Introduction
  2. The MPC-MC Architecture
  3. Deterministic and Stochastic Variants of MPC-MC
  4. Computational Results
  5. MPC-MC with Stockfish and Komodo Dragon Chess Engines
  6. Comparison with a Half-Step Lookahead Version of MPC-MC
  7. MPC-MC with Multistep Lookahead
  8. Concluding Remarks

Figures (3)

  • Figure 1: Schematic illustration of the MPC-MC framework with one-step lookahead. The position evaluator and the nominal opponent in the figure are SK engines.
  • Figure 2: Schematic illustration of the MPC-MC framework with half-step lookahead. Here the SK engine is used to evaluate (from the opponent's point of view) all legal moves at the current position. Then MPC-MC selects the move that is most difficult for the opponent, i.e., the one that results in the worst evaluation according to SK.
  • Figure 3: Schematic illustration of the MPC-MC framework with two-step lookahead, and a deterministic nominal opponent. The position evaluator and the nominal opponent in the figure are SK engines.