Are Multimodal Large Language Models Ready for Omnidirectional Spatial Reasoning?

TL;DR

This work introduces OSR-Bench, the first benchmark focused on omnidirectional spatial reasoning for Multimodal LLMs using 360° panoramic indoor scenes. It builds OSR-Dataset from high-fidelity panoramic layouts, generates over 153k QA pairs across object counting, relative distance, and relative direction, and employs a two-stage evaluation with rotation-invariant cognitive-map matching and both rule-based and LLM-based QA assessment, including negative sampling to probe hallucination. An evaluation of eight state-of-the-art MLLMs in zero-shot settings reveals pronounced gaps in panoramic spatial reasoning, with proprietary models generally outperforming open-source ones and cognitive maps yielding mixed benefits across tasks. The results highlight the need for perceptually grounded MLLMs capable of robust grounding and reasoning in panoramic contexts, and the authors provide OSR-Bench and code for community use.

Abstract

The 180x360 omnidirectional field of view captured by 360-degree cameras enables their use in a wide range of applications such as embodied AI and virtual reality. Although recent advances in multimodal large language models (MLLMs) have shown promise in visual-spatial reasoning, most studies focus on standard pinhole-view images, leaving omnidirectional perception largely unexplored. In this paper, we ask: Are MLLMs ready for omnidirectional spatial reasoning? To investigate this, we introduce OSR-Bench, the first benchmark specifically designed for this setting. OSR-Bench includes over 153,000 diverse question-answer pairs grounded in high-fidelity panoramic indoor scene maps. It covers key reasoning types including object counting, relative distance, and direction. We also propose a negative sampling strategy that inserts non-existent objects into prompts to evaluate hallucination and grounding robustness. For fine-grained analysis, we design a two-stage evaluation framework assessing both cognitive map generation and QA accuracy using rotation-invariant matching and a combination of rule-based and LLM-based metrics. We evaluate eight state-of-the-art MLLMs, including GPT-4o, Gemini 1.5 Pro, and leading open-source models under zero-shot settings. Results show that current models struggle with spatial reasoning in panoramic contexts, highlighting the need for more perceptually grounded MLLMs. OSR-Bench and code will be released at: https://huggingface.co/datasets/UUUserna/OSR-Bench

Figures (14)

• Figure 1: Overview of our Omni-Spatial-Reasoning (OSR) dataset and benchmark.
• Figure 2: Cases of the established Omni-Spatial-Reasoning dataset.
• Figure 3: The data generation pipeline of our dataset.
• Figure 4: Benchmark Evaluation Pipeline.
• Figure 5: Qualitative examples of omnidirectional spatial reasoning on our OSR-dataset.