R4: Retrieval-Augmented Reasoning for Vision-Language Models in 4D Spatio-Temporal Space

TL;DR

R4 presents a training-free framework that equips vision-language models with a lifelong, structured 4D memory anchored in a global SLAM map. By decomposing natural language queries into semantic, spatial, and temporal keys, it performs retrieval-augmented reasoning over persistent observations, enabling long-horizon, embodied, and multi-agent reasoning without fine-tuning. The approach yields state-of-the-art results on embodied reasoning benchmarks (ERQA/OpenEQA) and demonstrates effective collaboration through shared 4D memory, illustrating a scalable path toward human-like, 4D cognitive priors in dynamic environments. This work introduces a new paradigm for embodied vision-language intelligence by grounding reasoning in a continuously updated, queryable world model.

Abstract

Humans perceive and reason about their surroundings in four dimensions by building persistent, structured internal representations that encode semantic meaning, spatial layout, and temporal dynamics. These multimodal memories enable them to recall past events, infer unobserved states, and integrate new information into context-dependent reasoning. Inspired by this capability, we introduce R4, a training-free framework for retrieval-augmented reasoning in 4D spatio-temporal space that equips vision-language models (VLMs) with structured, lifelong memory. R4 continuously constructs a 4D knowledge database by anchoring object-level semantic descriptions in metric space and time, yielding a persistent world model that can be shared across agents. At inference, natural language queries are decomposed into semantic, spatial, and temporal keys to retrieve relevant observations, which are integrated into the VLM's reasoning. Unlike classical retrieval-augmented generation methods, retrieval in R4 operates directly in 4D space, enabling episodic and collaborative reasoning without training. Experiments on embodied question answering and navigation benchmarks demonstrate that R4 substantially improves retrieval and reasoning over spatio-temporal information compared to baselines, advancing a new paradigm for embodied 4D reasoning in dynamic environments.

Figures (8)

• Figure 1: Overview of the R$^4$ framework. R$^4$ couples continuous storage with a structured retrieval-reasoning loop in a consistent 4D knowledge database. Object-level features are anchored in global coordinates and shared across agents to provide spatio-temporal context. This enables vision-language models to reason in four dimensions over long horizons for embodied tasks without retraining.
• Figure 2: Storage pipeline: generation of object-level 4D features and insertion into the continuous 4D knowledge database.
• Figure 3: Retrieval-augmented 4D reasoning pipeline. Queries that cannot be answered from live perception trigger retrieval-augmented reasoning over semantic, spatial, and temporal keys. Retrieved context is re-injected into the VLM for reasoning.
• Figure 4: Capability profile on ERQA. Radar plot illustrating the performance of R$^4$ across eight core reasoning dimensions: action reasoning, spatial reasoning, other, pointing, multi-view reasoning, task reasoning, state estimation, and trajectory reasoning.
• Figure 5: Qualitative examples on ERQA. Each example displays the corresponding visual observations as vertical frame sequences alongside its question and predicted answer in Table \ref{['tab:erqa_examples_appendix']}. The top block illustrates correct cases, while the bottom block highlights representative failure modes.