Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

StateLinFormer: Stateful Training Enhancing Long-term Memory in Navigation

Zhiyuan Chen, Yuxuan Zhong, Fan Wang, Bo Yu, Pengtao Shao, Shaoshan Liu, Ning Ding

Abstract

Effective navigation intelligence relies on long-term memory to support both immediate generalization and sustained adaptation. However, existing approaches face a dilemma: modular systems rely on explicit mapping but lack flexibility, while Transformer-based end-to-end models are constrained by fixed context windows, limiting persistent memory across extended interactions. We introduce StateLinFormer, a linear-attention navigation model trained with a stateful memory mechanism that preserves recurrent memory states across consecutive training segments instead of reinitializing them at each batch boundary. This training paradigm effectively approximates learning on infinitely long sequences, enabling the model to achieve long-horizon memory retention. Experiments across both MAZE and ProcTHOR environments demonstrate that StateLinFormer significantly outperforms its stateless linear-attention counterpart and standard Transformer baselines with fixed context windows. Notably, as interaction length increases, persistent stateful training substantially improves context-dependent adaptation, suggesting an enhancement in the model's In-Context Learning (ICL) capabilities for navigation tasks.

StateLinFormer: Stateful Training Enhancing Long-term Memory in Navigation

Abstract

Effective navigation intelligence relies on long-term memory to support both immediate generalization and sustained adaptation. However, existing approaches face a dilemma: modular systems rely on explicit mapping but lack flexibility, while Transformer-based end-to-end models are constrained by fixed context windows, limiting persistent memory across extended interactions. We introduce StateLinFormer, a linear-attention navigation model trained with a stateful memory mechanism that preserves recurrent memory states across consecutive training segments instead of reinitializing them at each batch boundary. This training paradigm effectively approximates learning on infinitely long sequences, enabling the model to achieve long-horizon memory retention. Experiments across both MAZE and ProcTHOR environments demonstrate that StateLinFormer significantly outperforms its stateless linear-attention counterpart and standard Transformer baselines with fixed context windows. Notably, as interaction length increases, persistent stateful training substantially improves context-dependent adaptation, suggesting an enhancement in the model's In-Context Learning (ICL) capabilities for navigation tasks.
Paper Structure (19 sections, 13 equations, 4 figures, 3 tables)

This paper contains 19 sections, 13 equations, 4 figures, 3 tables.

Table of Contents

  1. INTRODUCTION
  2. RELATED WORK
  3. Explicit Mapping-Based Navigation
  4. Transformer Models for Navigation
  5. Memory-Based Adaptation
  6. Methodologies
  7. Problem Formulation
  8. Model Architecture
  9. Stateful Training of linear Attention with long-term Memory
  10. Optimization Perspective
  11. Experiments
  12. Experimental setup
  13. Maze
  14. ProcTHOR
  15. Empirical Evidence for Stable Memory Regimes.
  16. ...and 4 more sections

Figures (4)

  • Figure 1: The framework of StateLinFormer. Top: Conventional stateless training with memory reset each batch. Bottom: StateLinFormer, which maintains persistent memory across batches to support long-horizon adaptation.
  • Figure 2: Relative Standard Deviation (RSD) of Memory Norms: Stateful vs. Stateless Training.
  • Figure 3: Comparison of Success Rates between Stateful and Stateless Training across Increasing Context Lengths
  • Figure 4: The model structure of StateLinformer