Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Transfer in Deep Reinforcement Learning using Knowledge Graphs

Prithviraj Ammanabrolu, Mark O. Riedl

TL;DR

Problem: transferring control policies across different text-adventure games is challenging due to partial observability and a combinatorially large action space. Approach: the authors extend KG-DQN by using knowledge graphs as state representations, seeding graphs from static textual guides, pre-training parts of the network with a QA system, and transferring network weights between games in the same genre. Findings: the combined transfer methods yield faster convergence and higher-quality policies, with up to about 80% gains in completion steps and statistically significant improvements (p<0.05). Significance: this work demonstrates a practical pathway to ground language-driven agents in structured knowledge graphs, enabling more efficient cross-task transfer in language-based RL.

Abstract

Text adventure games, in which players must make sense of the world through text descriptions and declare actions through text descriptions, provide a stepping stone toward grounding action in language. Prior work has demonstrated that using a knowledge graph as a state representation and question-answering to pre-train a deep Q-network facilitates faster control policy transfer. In this paper, we explore the use of knowledge graphs as a representation for domain knowledge transfer for training text-adventure playing reinforcement learning agents. Our methods are tested across multiple computer generated and human authored games, varying in domain and complexity, and demonstrate that our transfer learning methods let us learn a higher-quality control policy faster.

Transfer in Deep Reinforcement Learning using Knowledge Graphs

TL;DR

Problem: transferring control policies across different text-adventure games is challenging due to partial observability and a combinatorially large action space. Approach: the authors extend KG-DQN by using knowledge graphs as state representations, seeding graphs from static textual guides, pre-training parts of the network with a QA system, and transferring network weights between games in the same genre. Findings: the combined transfer methods yield faster convergence and higher-quality policies, with up to about 80% gains in completion steps and statistically significant improvements (p<0.05). Significance: this work demonstrates a practical pathway to ground language-driven agents in structured knowledge graphs, enabling more efficient cross-task transfer in language-based RL.

Abstract

Text adventure games, in which players must make sense of the world through text descriptions and declare actions through text descriptions, provide a stepping stone toward grounding action in language. Prior work has demonstrated that using a knowledge graph as a state representation and question-answering to pre-train a deep Q-network facilitates faster control policy transfer. In this paper, we explore the use of knowledge graphs as a representation for domain knowledge transfer for training text-adventure playing reinforcement learning agents. Our methods are tested across multiple computer generated and human authored games, varying in domain and complexity, and demonstrate that our transfer learning methods let us learn a higher-quality control policy faster.

Paper Structure

This paper contains 12 sections, 2 equations, 6 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Background and Related Work
  3. Knowledge Graphs for DQNs
  4. Knowledge Graph Seeding
  5. Task Specific Transfer
  6. Experiments
  7. Slice of Life Experiments
  8. Horror Experiments
  9. Reward Augmentation
  10. Results/Discussion
  11. Conclusions
  12. Acknowledgements

Figures (6)

  • Figure 1: The KG-DQN architecture.
  • Figure 2: Select partial example of what a seed knowledge graph looks like. Ellipses indicate other similar entities and relations not shown.
  • Figure 3: Partial unseeded knowledge graph example given observations and actions in the game 9:05.
  • Figure 4: Partial unseeded knowledge graph example given observations and actions in the game Anchorhead.
  • Figure 5: Reward curve for select experiments in the slice of life domain.
  • ...and 1 more figures