SpatialPrompting: Keyframe-driven Zero-Shot Spatial Reasoning with Off-the-Shelf Multimodal Large Language Models

TL;DR

SpatialPrompting addresses zero-shot spatial reasoning in 3D by eschewing expensive 3D-specific training and instead generating keyframe-driven prompts for off-the-shelf multimodal LLMs. The framework selects a compact set of informative keyframes using a joint metric that combines spatial diversity and semantic content, quantified as $d'(i, j) = d(i, j) + \alpha(1 - S(i, j))$, along with a quality score $q_t = \det|\Sigma_t| + \beta \operatorname{Var}(\nabla^2 I_t)$, then feeds keyframes with camera poses into prompts composed of a preamble, pose-annotated frames, an annotation, and a user query. Experiments on ScanQA and SQA3D show SpatialPrompting achieves state-of-the-art zero-shot performance on key metrics like EM@1, ROUGE-L, and SPICE on ScanQA and strong results on SQA3D categories, without 3D inputs or fine-tuning. The approach is efficient and scalable, leveraging existing multimodal LLMs to perform complex spatial reasoning, with potential impact on indoor robotics, AR, and human–computer interaction.

Abstract

This study introduces SpatialPrompting, a novel framework that harnesses the emergent reasoning capabilities of off-the-shelf multimodal large language models to achieve zero-shot spatial reasoning in three-dimensional (3D) environments. Unlike existing methods that rely on expensive 3D-specific fine-tuning with specialized 3D inputs such as point clouds or voxel-based features, SpatialPrompting employs a keyframe-driven prompt generation strategy. This framework uses metrics such as vision-language similarity, Mahalanobis distance, field of view, and image sharpness to select a diverse and informative set of keyframes from image sequences and then integrates them with corresponding camera pose data to effectively abstract spatial relationships and infer complex 3D structures. The proposed framework not only establishes a new paradigm for flexible spatial reasoning that utilizes intuitive visual and positional cues but also achieves state-of-the-art zero-shot performance on benchmark datasets, such as ScanQA and SQA3D, across several metrics. The proposed method effectively eliminates the need for specialized 3D inputs and fine-tuning, offering a simpler and more scalable alternative to conventional approaches.

Figures (11)

• Figure 1: SpatialPrompting is a framework that employs keyframe-driven prompt generation for zero-shot spatial reasoning with multimodal LLMs. The proposed approach enables accurate spatial reasoning without additional 3D-specific training.
• Figure 2: Overview of SpatialPrompting. In keyframe extraction, both spatial and semantic features are used to select keyframes. In prompt generation, these keyframes and camera poses are combined with a preamble, annotation, and user query to form prompts for multimodal LLMs, enabling SpatialQA.
• Figure 3: Qualitative results. We showcase five categories of spatial reasoning tasks tackled by the proposed SpatialPrompting framework: (1) Small object localization, (2) Complex spatial layout inference, (3) Multi-object relationship reasoning, (4) Relative spatial orientation, and (5) Real-world knowledge integration. Each question--answer pair is generated using minimal keyframes and corresponding camera poses, illustrating how the proposed method handles a range of scenarios---from pinpointing small items to considering practical constraints (e.g., vehicle capacity for transporting a piano).
• Figure 4: Effectiveness of Keyframe extraction on SpatialPrompting. The proposed keyframe extraction process effectively eliminates redundant views, and the results indicate that images with a wider FOV are preferentially selected.
• Figure 5: Effectiveness of SpatialPrompting in limited images. The use of high-coverage keyframes reduces missing information, whereas camera poses allow the inference of spatial relationships from limited images, enabling the proposed method to achieve complex SpatialQA.