See, Remember, Explore: A Benchmark and Baselines for Streaming Spatial Reasoning

Abstract

Spatial understanding is fundamental for embodied agents, yet most spatial VLMs and benchmarks remain offline-evaluating post-hoc QA over pre-recorded inputs and overlooking two crucial deployment-critical requirements: long-horizon streaming inference and active perception when the current view is insufficient. To address this gap, we introduce S3-Bench, a benchmark suite for streaming spatial question answering with active exploration, where queries are temporally grounded to specific timestamps and must be answered using only observations available up to that moment. S3-Bench adopts a dual-domain design, combining a scalable simulator with controllable trajectories and exploration actions, and real-world streaming videos that capture practical sensing artifacts for rigorous generalization evaluation. Overall, it spans 10K+ scenes and 26K+ trajectories, with dedicated training (S3-Train) and evaluation (S3-Eval) splits. We further propose AMF-VLM, which supports streaming spatial reasoning under bounded computing via (i) memory folding, which compresses long-horizon observations into compact structured memory, and (ii) active exploration, which outputs explicit actions (e.g. move/rotate/scan) to acquire missing evidence before answering. Extensive experiments demonstrate that, compared to models using identical training data, our approach yields improvements of 8.8% and 13.3% on the simulated and real splits of S3-Eval, respectively, while maintaining competitive transferability to standard spatial benchmarks.

Figures (9)

• Figure 1: Overview of the $S^3$-Bench and AMF-VLM. Unlike traditional offline methods that answer questions post-video, our $S^3$-Bench targets streaming spatial understanding with timestamp-specific, ego-centric, and temporally dependent queries. The figure outlines the AMF-VLM pipeline and presents its performance comparison with baselines on $S^3$-Eval.
• Figure 2: $S^3$ Benchmark data construction pipeline. Simulated digital assets are filtered and rendered into videos via geometry-guided path planning. Next, object bounding boxes and frame-level visibility are extracted to generate template-based QA pairs, which are then rewritten for linguistic diversity.
• Figure 3: Distribution and examples of QA categories in the $S^3$-Bench. This figure illustrates the statistical distribution of different QA categories alongside representative examples. The benchmark primarily comprises three question types: (1) Ego-centric, requiring understanding based on the current camera position; (2) Spatial relation, assessing the comprehension of the scene's spatial information; and (3) Temporal dependency, involving streaming queries that rely on preceding context.
• Figure 4: AMF-VLM architecture. Visual features are extracted via a vision encoder, optionally supplemented by a spatial encoder. For efficient streaming, recent frames are densely sampled while long-term history is sparsely retained and compressed via memory folding. Finally, the model outputs either an exploration action, a direct answer, or a memory update summary.
• Figure 5: Qualitative examples of Active Exploration. Restricted initial views can lead to incorrect direct predictions. To mitigate this, the model executes exploration actions to gather sufficient visual context before responding.