Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

A Machine with Short-Term, Episodic, and Semantic Memory Systems

Taewoon Kim, Michael Cochez, Vincent François-Lavet, Mark Neerincx, Piek Vossen

TL;DR

This work modeled an agent with short-term, episodic, and semantic memory systems, each of which is modeled with a knowledge graph, and shows that this deep Q-learning based agent successfully learns whether a long-term memory should be forgotten, or rather be stored in the episodic or semantic memory system.

Abstract

Inspired by the cognitive science theory of the explicit human memory systems, we have modeled an agent with short-term, episodic, and semantic memory systems, each of which is modeled with a knowledge graph. To evaluate this system and analyze the behavior of this agent, we designed and released our own reinforcement learning agent environment, "the Room", where an agent has to learn how to encode, store, and retrieve memories to maximize its return by answering questions. We show that our deep Q-learning based agent successfully learns whether a short-term memory should be forgotten, or rather be stored in the episodic or semantic memory systems. Our experiments indicate that an agent with human-like memory systems can outperform an agent without this memory structure in the environment.

A Machine with Short-Term, Episodic, and Semantic Memory Systems

TL;DR

This work modeled an agent with short-term, episodic, and semantic memory systems, each of which is modeled with a knowledge graph, and shows that this deep Q-learning based agent successfully learns whether a long-term memory should be forgotten, or rather be stored in the episodic or semantic memory system.

Abstract

Inspired by the cognitive science theory of the explicit human memory systems, we have modeled an agent with short-term, episodic, and semantic memory systems, each of which is modeled with a knowledge graph. To evaluate this system and analyze the behavior of this agent, we designed and released our own reinforcement learning agent environment, "the Room", where an agent has to learn how to encode, store, and retrieve memories to maximize its return by answering questions. We show that our deep Q-learning based agent successfully learns whether a short-term memory should be forgotten, or rather be stored in the episodic or semantic memory systems. Our experiments indicate that an agent with human-like memory systems can outperform an agent without this memory structure in the environment.
Paper Structure (17 sections, 2 equations, 7 figures, 2 tables, 2 algorithms)

This paper contains 17 sections, 2 equations, 7 figures, 2 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Methodology
  3. The Room Environment
  4. Human-Like Memory Systems
  5. Memory Storage
  6. Memory Retrieval
  7. The Deep Q-learning Reinforcement Learning Agent
  8. Experimental Setup
  9. The Environment and Agent Parameters
  10. The Deep Q-learning and Other Training Parameters
  11. Results
  12. Deep Q-learning Performance by Agent
  13. Understanding What the Agent Learns
  14. Related Work
  15. Conclusions
  16. ...and 2 more sections

Figures (7)

  • Figure 1: The memory systems of the agent. The long-term (explicit) memory systems consist of episodic and semantic memory systems.
  • Figure 2: An episodic memory and a semantic memory represented as a knowledge graph.
  • Figure 3: The Q-network diagram, where the short-term $\bm{M}_{o}$, episodic $\bm{M}_{e}$, and semantic $\bm{M}_{s}$ memory systems are given as the initial input. The LSTM blocks output the last hidden states of the LSTMs. The final output is the state-action values of the $Q$ function.
  • Figure 4: An example of converting an episodic memory system into a KGE representation.
  • Figure 5: Training, validation, and test results of the agents with the memory capacity of 32. All of them were run five times with different random seeds. The standard deviation is shown together with the average. As for training, instead of total rewards per episode, we report loss over time since the varying epsilon value affects the rewards. The maximum total rewards per episode are 128, since in total of 128 questions are asked per episode.
  • ...and 2 more figures