Playing a Strategy Game with Knowledge-Based Reinforcement Learning

TL;DR

The reported experiment supports the idea that, based on human knowledge and empowered by RL, the KB-RL system can deliver a strong solution to the complex, multi-strategic problems, and, mainly, improve the solution with increased experience.

Abstract

This paper presents Knowledge-Based Reinforcement Learning (KB-RL) as a method that combines a knowledge-based approach and a reinforcement learning (RL) technique into one method for intelligent problem solving. The proposed approach focuses on multi-expert knowledge acquisition, with the reinforcement learning being applied as a conflict resolution strategy aimed at integrating the knowledge of multiple exerts into one knowledge base. The article describes the KB-RL approach in detail and applies the reported method to one of the most challenging problems of current Artificial Intelligence (AI) research, namely playing a strategy game. The results show that the KB-RL system is able to play and complete the full FreeCiv game, and to win against the computer players in various game settings. Moreover, with more games played, the system improves the gameplay by shortening the number of rounds that it takes to win the game. Overall, the reported experiment supports the idea that, based on human knowledge and empowered by reinforcement learning, the KB-RL system can deliver a strong solution to the complex, multi-strategic problems, and, mainly, to improve the solution with increased experience.

Figures (12)

• Figure 1: An example of the Issue object as a holder for working memory in KB-RL system. The example shows the content of the Issue object on its creation. As more knowledge is applied to the problem, the Issue object will hold all relevant content in new attribute/value pairs.
• Figure 2: An example of the Knowledge Item as a rule in KB-RL.
• Figure 3: Ontology of the FreeCiv game as the representation of the game model for the semantic network within our KB-RL system. For the sake of simplicity, the properties lists are not exhaustive, but rather illustrative.
• Figure 4: Map topology used in the game setups. a) Map topology for Default, Small Islands and USA setups; b) Map topology for Chaos setup; c) Map topology for Medium Islands setup. The color marks the terrain type of the tile.
• Figure 5: The probabilities for the KIs (actions) of the conflict set derived as an area under the curve on the right to the limit line L. The KI 1 has the highest action-value, thus $\mu$ and $\sigma$ parameters calculated for KI 1 are taken to form the limit $L=\mu_{max}-\sigma_{max}$.