Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

A Minimal Task Reveals Emergent Path Integration and Object-Location Binding in a Predictive Sequence Model

Linda Ariel Ventura, Victoria Bosch, Tim C Kietzmann, Sushrut Thorat

TL;DR

This work investigates how structured world models can emerge from action-conditioned sequence prediction in a minimal 2D token scene. A 3-layer GRU predicts the next token given the current token and a saccade-like displacement, and the model exhibits in-context learning on novel scenes, with decoding analyses revealing path integration and token–position binding as core mechanisms. Interventional experiments show that new label–position bindings can be learned late in a sequence and generalized to out-of-distribution positions, suggesting retrieval-based binding rather than a static dictionary. The findings provide a mechanistic, minimal framework for understanding how sequential prediction can instantiate a structured world model and offer insights into potential cognitive-science and active-perception parallels.

Abstract

Adaptive cognition requires structured internal models representing objects and their relations. Predictive neural networks are often proposed to form such "world models", yet their underlying mechanisms remain unclear. One hypothesis is that action-conditioned sequential prediction suffices for learning such world models. In this work, we investigate this possibility in a minimal in-silico setting. Sequentially sampling tokens from 2D continuous token scenes, a recurrent neural network is trained to predict the upcoming token from current input and a saccade-like displacement. On novel scenes, prediction accuracy improves across the sequence, indicating in-context learning. Decoding analyses reveal path integration and dynamic binding of token identity to position. Interventional analyses show that new bindings can be learned late in sequence and that out-of-distribution bindings can be learned. Together, these results demonstrate how structured representations that rely on flexible binding emerge to support prediction, offering a mechanistic account of sequential world modeling relevant to cognitive science.

A Minimal Task Reveals Emergent Path Integration and Object-Location Binding in a Predictive Sequence Model

TL;DR

This work investigates how structured world models can emerge from action-conditioned sequence prediction in a minimal 2D token scene. A 3-layer GRU predicts the next token given the current token and a saccade-like displacement, and the model exhibits in-context learning on novel scenes, with decoding analyses revealing path integration and token–position binding as core mechanisms. Interventional experiments show that new label–position bindings can be learned late in a sequence and generalized to out-of-distribution positions, suggesting retrieval-based binding rather than a static dictionary. The findings provide a mechanistic, minimal framework for understanding how sequential prediction can instantiate a structured world model and offer insights into potential cognitive-science and active-perception parallels.

Abstract

Adaptive cognition requires structured internal models representing objects and their relations. Predictive neural networks are often proposed to form such "world models", yet their underlying mechanisms remain unclear. One hypothesis is that action-conditioned sequential prediction suffices for learning such world models. In this work, we investigate this possibility in a minimal in-silico setting. Sequentially sampling tokens from 2D continuous token scenes, a recurrent neural network is trained to predict the upcoming token from current input and a saccade-like displacement. On novel scenes, prediction accuracy improves across the sequence, indicating in-context learning. Decoding analyses reveal path integration and dynamic binding of token identity to position. Interventional analyses show that new bindings can be learned late in sequence and that out-of-distribution bindings can be learned. Together, these results demonstrate how structured representations that rely on flexible binding emerge to support prediction, offering a mechanistic account of sequential world modeling relevant to cognitive science.
Paper Structure (10 sections, 4 figures, 1 algorithm)

This paper contains 10 sections, 4 figures, 1 algorithm.

Table of Contents

  1. Introduction
  2. Methods
  3. Minimal scene construction
  4. Network architecture & training
  5. Results
  6. In-context learning of token arrangement in scenes
  7. Signatures of path integration and token label-position binding
  8. The stability, plasticity, and generalizability of in-context scene memory
  9. Discussion
  10. Acknowledgments

Figures (4)

  • Figure 1: A. Conceptual figure depicting action-conditioned prediction as a part of behaving agents. B. Prediction network architecture and example of a scene with $6$ tokens placed in a $2$D continuous space. The network receives the saccade displacement vector and the label of the current token, and is tasked to output the label of the subsequent token in the sequence. C. Average model prediction performance in unseen worlds (N=$500$) per sequence timestep. Error envelope depicts 95% CIs of the means.
  • Figure 2: A. Example pentagonal scene used for the decoding analyses. We generate 500 test scenes with this arrangement. B. Decoding of token label and absolute token position across model layers, averaged across timesteps 35-100. C. Decoding of label-position tuples across model layers, averaged across timesteps 35-100, compared to their corresponding baselines: expected decoding accuracy based on the decoding of the components (product of their decoding accuracies). The top panels show results for congruent label-position pairs, and the bottom panels depict results for incongruent label-position pairs. We highlight the results for which the baseline accuracy is below 1, as in the cases of perfect component decoding we cannot meaningfully measure accuracy deviations that would indicate binding. Error bars depict 95% CIs of the means.
  • Figure 3: A. Left: Illustration of the token change procedure. Middle: Performance under intervention for tokens at the changed and unchanged locations (N=500 scenes). Right: Error contribution of original replaced labels and other labels at other locations. B. Left: Illustration of the token addition procedure. Middle: Model performance after token addition at timestep 35 (N=500 scenes). Right: Model performance after token addition at timestep 100 (N=500 scenes). Error envelope depicts 95% CIs of the means.
  • Figure 4: A. Illustration of tuple generalization analysis: during training, the token k is only shown at position (1,1) while other tokens can be placed anywhere on the grid. B. Model performance on test scenes with a non-k token placed at the control location (1,1), and k placed in the other three quadrants (N=500 scenes). Error envelope depicts 95% CIs of the means.