Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

A transformer-based deep reinforcement learning approach to spatial navigation in a partially observable Morris Water Maze

Marte Eggen, Inga Strümke

TL;DR

This work applies a transformer-based architecture using deep reinforcement learning – an approach previously unexplored in this context – to navigate a 2D version of the Morris Water Maze, demonstrating the potential of transformer-based models for enhancing navigation performance in partially observable environments.

Abstract

Navigation is a fundamental cognitive skill extensively studied in neuroscientific experiments and has lately gained substantial interest in artificial intelligence research. Recreating the task solved by rodents in the well-established Morris Water Maze (MWM) experiment, this work applies a transformer-based architecture using deep reinforcement learning -- an approach previously unexplored in this context -- to navigate a 2D version of the maze. Specifically, the agent leverages a decoder-only transformer architecture serving as a deep Q-network performing effective decision making in the partially observable environment. We demonstrate that the proposed architecture enables the agent to efficiently learn spatial navigation strategies, overcoming challenges associated with a limited field of vision, corresponding to the visual information available to a rodent in the MWM. Demonstrating the potential of transformer-based models for enhancing navigation performance in partially observable environments, this work suggests promising avenues for future research in artificial agents whose behavior resembles that of biological agents. Finally, the flexibility of the transformer architecture in supporting varying input sequence lengths opens opportunities for gaining increased understanding of the artificial agent's inner representation of the environment.

A transformer-based deep reinforcement learning approach to spatial navigation in a partially observable Morris Water Maze

TL;DR

This work applies a transformer-based architecture using deep reinforcement learning – an approach previously unexplored in this context – to navigate a 2D version of the Morris Water Maze, demonstrating the potential of transformer-based models for enhancing navigation performance in partially observable environments.

Abstract

Navigation is a fundamental cognitive skill extensively studied in neuroscientific experiments and has lately gained substantial interest in artificial intelligence research. Recreating the task solved by rodents in the well-established Morris Water Maze (MWM) experiment, this work applies a transformer-based architecture using deep reinforcement learning -- an approach previously unexplored in this context -- to navigate a 2D version of the maze. Specifically, the agent leverages a decoder-only transformer architecture serving as a deep Q-network performing effective decision making in the partially observable environment. We demonstrate that the proposed architecture enables the agent to efficiently learn spatial navigation strategies, overcoming challenges associated with a limited field of vision, corresponding to the visual information available to a rodent in the MWM. Demonstrating the potential of transformer-based models for enhancing navigation performance in partially observable environments, this work suggests promising avenues for future research in artificial agents whose behavior resembles that of biological agents. Finally, the flexibility of the transformer architecture in supporting varying input sequence lengths opens opportunities for gaining increased understanding of the artificial agent's inner representation of the environment.

Paper Structure

This paper contains 8 sections, 4 equations, 2 figures.

Table of Contents

  1. Introduction
  2. Background
  3. The transformer architecture
  4. Deep Q-networks
  5. Method
  6. Results and analysis
  7. Discussion
  8. Conclusion

Figures (2)

  • Figure 1: The simulated 2D MWM environment includes a visual cue, represented by the red portion of the circumference, and the agent's field of vision comprises 12 distinct sight lines spanning a fixed angle. The navigation objective is to reach the hidden circular platform. The scenarios illustrate (a) the trained agent initially following a direct trajectory towards the center and (b) subsequently rotating until observing the visual cue to adjust the navigation path towards the platform. The smaller dots indicate the steps taken, whereas the larger dot at the front of the line represents the agent's current position.
  • Figure 2: Exponential moving average of (a) the total reward and (b) the number of steps per episode across five independent runs for input sequence lengths of 5, 45, and 75 observations.