Constructing coherent spatial memory in LLM agents through graph rectification

TL;DR

The paper tackles the problem of maintaining coherent spatial memory in long-horizon text-based navigation by enabling LLM-driven, incremental graph construction with robust, history-aware repair. It introduces LLM-MapRepair, a modular framework comprising conflict detection, error localization, and a version-controlled graph history, augmented by an Edge Impact Score to prioritize low-risk repairs and mitigate cascading inconsistencies. A formal localization pipeline (minimal conflicting path pairs, LCA, candidate edge extraction, and scoring) and a CRG-based prioritization objective drive targeted, interpretable repairs, with a commit-based Version Control system enabling rollback and causal tracing. Experiments on a refined MANGO dataset show that combining Version Control with Edge-Impact Ranking improves repair efficiency and robustness, especially against long-range error propagation, demonstrating the value of introspective, history-aware mechanisms for maintaining coherent spatial memory in LLM agents. The work advances practical spatial reasoning in language-driven agents and informs future knowledge-graph–based navigation systems by integrating incremental mapping with explicit error management.

Abstract

Given a map description through global traversal navigation instructions (e.g., visiting each room sequentially with action signals such as north, west, etc.), an LLM can often infer the implicit spatial layout of the environment and answer user queries by providing a shortest path from a start to a destination (for instance, navigating from the lobby to a meeting room via the hall and elevator). However, such context-dependent querying becomes incapable as the environment grows much longer, motivating the need for incremental map construction that builds a complete topological graph from stepwise observations. We propose a framework for LLM-driven construction and map repair, designed to detect, localize, and correct structural inconsistencies in incrementally constructed navigation graphs. Central to our method is the Version Control, which records the full history of graph edits and their source observations, enabling fine-grained rollback, conflict tracing, and repair evaluation. We further introduce an Edge Impact Score to prioritize minimal-cost repairs based on structural reachability, path usage, and conflict propagation. To properly evaluate our approach, we create a refined version of the MANGO benchmark dataset by systematically removing non-topological actions and inherent structural conflicts, providing a cleaner testbed for LLM-driven construction and map repair. Our approach significantly improves map correctness and robustness, especially in scenarios with entangled or chained inconsistencies. Our results highlight the importance of introspective, history-aware repair mechanisms for maintaining coherent spatial memory in LLM agents.

Figures (6)

• Figure 1: Spatial reasoning paradigms for LLM agents: Direct context-based approach with inherent limitations versus incremental graph construction with structured error handling capabilities.
• Figure 2: Overview of the LLM-MapRepair framework.
• Figure 3: (a) Three types of structural conflicts — naming conflict, directional conflict, and topological conflict. Highlighted nodes indicate the conflicting pairs. (b) A challenging conflict scenario: a misdirected edge from Room E to Room G introduces a latent spatial misalignment. A visible topology conflict emerges later between Room D and Room I (highlighted), while correcting the edge E$\rightarrow$G may trigger a new conflict between Room H and Room A. The edge E$\rightarrow$J is incorrect, though it is unrelated to the current conflict.
• Figure 4: Each commit $G_i$ represents a commit to the graph (e.g., edge additions or conflict-triggered replacements). The Version Control system maintains a linear commit history where each commit is indexed by its corresponding edge modification. Commit metadata includes the step identifier, specific edge changes (+ for additions, - for removals during replacements), trigger event type, and associated observation. The system supports three core operations: rollback to previous commits, recall of commit details, and diff comparisons between commits.
• Figure A1: Structural conflict distribution in MANGO environments. Blue segments represent topological conflicts; orange segments represent directional conflicts. Naming conflicts are not observed.