Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Extending a Quantum Reinforcement Learning Exploration Policy with Flags to Connect Four

Filipe Santos, João Paulo Fernandes, Luís Macedo

TL;DR

The paper extends Quantum Flagged Action Selection to Connect Four, integrating flag-based exploration with offline Deep Q-learning to handle the game's large state space. Against a Randomized Negamax opponent, it shows that flag-based exploration outperforms epsilon-greedy in training efficiency, and that the quantum variant reduces the iterations needed to obtain flagged actions, albeit with comparable win rates in the tested scenarios. The work demonstrates the generality of flag-based quantum exploration beyond Checkers and suggests potential gains in more complex settings or with stronger opponents. It also introduces a practical framework for encoding action probabilities on quantum hardware via nested coherent controlization, paving the way for broader application in sequential decision problems.

Abstract

Action selection based on flags is a Reinforcement Learning (RL) exploration policy that improves the exploration of the state space through the use of flags, which can identify the most promising actions to take in each state. The quantum counterpart of this exploration policy further improves upon this by taking advantage of a quadratic speedup for sampling flagged actions. This approach has already been successfully employed for the game of Checkers. In this work, we describe the application of this method to the context of Connect Four, in order to study its performance in a different setting, which can lead to a better generalization of the technique. We also kept track of a metric that wasn't taken into account in previous work: the average number of iterations to obtain a flagged action. Since going second is a significant disadvantage in Connect Four, we also had the intent of exploring how this more complex scenario would impact the performance of our approach. The experiments involved training and testing classical and quantum RL agents that played either going first or going second against a Randomized Negamax opponent. The results showed that both flagged exploration policies were clearly superior to a simple epsilon-greedy policy. Furthermore, the quantum agents did in fact sample flagged actions in less iterations. Despite obtaining tagged actions more consistently, the win rates between the classical and quantum versions of the approach were identical, which could be due to the simplicity of the training scenario chosen.

Extending a Quantum Reinforcement Learning Exploration Policy with Flags to Connect Four

TL;DR

The paper extends Quantum Flagged Action Selection to Connect Four, integrating flag-based exploration with offline Deep Q-learning to handle the game's large state space. Against a Randomized Negamax opponent, it shows that flag-based exploration outperforms epsilon-greedy in training efficiency, and that the quantum variant reduces the iterations needed to obtain flagged actions, albeit with comparable win rates in the tested scenarios. The work demonstrates the generality of flag-based quantum exploration beyond Checkers and suggests potential gains in more complex settings or with stronger opponents. It also introduces a practical framework for encoding action probabilities on quantum hardware via nested coherent controlization, paving the way for broader application in sequential decision problems.

Abstract

Action selection based on flags is a Reinforcement Learning (RL) exploration policy that improves the exploration of the state space through the use of flags, which can identify the most promising actions to take in each state. The quantum counterpart of this exploration policy further improves upon this by taking advantage of a quadratic speedup for sampling flagged actions. This approach has already been successfully employed for the game of Checkers. In this work, we describe the application of this method to the context of Connect Four, in order to study its performance in a different setting, which can lead to a better generalization of the technique. We also kept track of a metric that wasn't taken into account in previous work: the average number of iterations to obtain a flagged action. Since going second is a significant disadvantage in Connect Four, we also had the intent of exploring how this more complex scenario would impact the performance of our approach. The experiments involved training and testing classical and quantum RL agents that played either going first or going second against a Randomized Negamax opponent. The results showed that both flagged exploration policies were clearly superior to a simple epsilon-greedy policy. Furthermore, the quantum agents did in fact sample flagged actions in less iterations. Despite obtaining tagged actions more consistently, the win rates between the classical and quantum versions of the approach were identical, which could be due to the simplicity of the training scenario chosen.
Paper Structure (12 sections, 12 equations, 3 figures, 2 tables)

This paper contains 12 sections, 12 equations, 3 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Literature Review
  3. A Randomized Negamax Opponent
  4. Reinforcement Learning for Connect Four
  5. Connect Four as a Markov Decision Process
  6. Dealing with the State Space Complexity
  7. The Exploration Policies
  8. Classical Flagged Action Selection
  9. Quantum Flagged Action Selection
  10. Encoding The Action Selection Probabilities
  11. Experimental Setup, Results and Discussion
  12. Conclusion

Figures (3)

  • Figure 1: A Markov chain for a specific state, with the transition probability to each action matching the probability of selecting that action when in that state. The flagged actions are colored orange.
  • Figure 2: A graph representing the transition probabilities for each action of a specific state, with each action corresponding to inserting a disc on that column. The flagged actions are colored orange.
  • Figure 3: The Inverse Temperature Functions.