Seeing from Another Perspective: Evaluating Multi-View Understanding in MLLMs

TL;DR

This paper introduces All-Angles Bench, a large-scale, real-world, multi-view benchmark designed to evaluate cross-view understanding in multi-modal large language models. It comprises over 2,100 questions across 90 scenes and six tasks that probe counting, attributes, relative distances/directions, object manipulation, and camera pose estimation, including paired questions to test cross-view consistency. Evaluations across 27 MLLMs reveal a substantial gap to human performance, with notable weaknesses in cross-view correspondence for occluded views and coarse camera pose estimation; some open-source models even outperform closed ones on orientation-sensitive tasks, underscoring the potential of domain-specific training. The authors provide a detailed construction and annotation pipeline, analyze prompting strategies, and argue for dedicated multi-view awareness modules and data to advance toward human-level multi-view understanding in embodied systems.

Abstract

Multi-view understanding, the ability to reconcile visual information across diverse viewpoints for effective navigation, manipulation, and 3D scene comprehension, is a fundamental challenge in Multi-Modal Large Language Models (MLLMs) to be used as embodied agents. While recent MLLMs have shown impressive advances in high-level reasoning and planning, they frequently fall short when confronted with multi-view geometric consistency and cross-view correspondence. To comprehensively evaluate the challenges of MLLMs in multi-view scene reasoning, we propose All-Angles Bench, a benchmark of over 2,100 human carefully annotated multi-view question-answer pairs across 90 diverse real-world scenes. Our six tasks (counting, attribute identification, relative distance, relative direction, object manipulation, and camera pose estimation) specifically test model's geometric correspondence and the capacity to align information consistently across views. Our extensive experiments, benchmark on 27 representative MLLMs including Gemini-2.0-Flash, Claude-3.7-Sonnet, and GPT-4o against human evaluators reveals a substantial performance gap, indicating that current MLLMs remain far from human-level proficiency. Through in-depth analysis, we show that MLLMs are particularly underperforming under two aspects: (1) cross-view correspondence for partially occluded views and (2) establishing the coarse camera poses. These findings highlight the necessity of domain-specific refinements or modules that embed stronger multi-view awareness. We believe that our All-Angles Bench offers valuable insights and contribute to bridging the gap between MLLMs and human-level multi-view understanding. The project and benchmark are publicly available at https://danielchyeh.github.io/All-Angles-Bench/.

Figures (29)

• Figure 1: Overview of All-Angles Bench. Our benchmark targets a comprehensive view of multi-view understanding, spanning six primary question types. These question types are designed to investigate several major aspects of 3D scene understanding, from creating correspondence between objects to associating relative object and camera poses.
• Figure 2: All-Angles Bench construction pipeline. (1) We collect and curate 90 diverse multi-view scenes and design six tasks that emphasize multi-view reasoning. (2) We generate initial questions via an MLLM, then refine and validate them through human annotation to ensure correctness, clarity, and domain relevance. (3) We create paired questions by systematically rephrasing or altering each view perspective while preserving their underlying visual correspondences to evaluate model's cross-view consistency. A final quality-control step removes inconsistent or ambiguous pairs. Note that counting and camera pose estimation tasks utilize all available views per query, whereas other tasks employ two randomly selected viewpoints.
• Figure 3: Statistical overview of All-Angles Bench. The pie chart shows the distribution of 6 sub-tasks of multi-view understanding. The bar plot illustrates the percentage breakdown by primary and paired question-answers of each sub-task.
• Figure 4: Evaluation results for 27 MLLMs. We consolidate performance from both closed-source and open-source MLLM evaluations. We use deeper-gray to highlight the top result among all models in each sub-task, while light-gray marks the second-best result.
• Figure 5: Paired question-answers inconsistency across 6 MLLMs. We report the proportions of IC and CC + WW. Notably, GPT-4o struggles with relative distance (around 70% inconsistency). Gemini-2.0-Flash and Claude-3.7-Sonnet exhibit more balanced performance, whereas Ovis2-34B and GPT-4o vary considerably across tasks.