PointArena: Probing Multimodal Grounding Through Language-Guided Pointing

TL;DR

PointArena presents a holistic benchmark for language-guided pointing that spans static localization (Point-Bench), human-preference ranking (Point-Battle), and real-world robotic execution (Point-Act). By formalizing pointing as a language-conditioned spatial localization task and incorporating ground-truth masks, the framework enables automated evaluation while preserving human-in-the-loop insight. Empirical results show that explicit pointing supervision markedly improves performance, and that correlations across Point-Bench, Point-Battle, and Point-Act validate PointArena as a reliable proxy for real-world grounding in multimodal models. The work highlights practical design choices for evaluation and offers a scalable, extensible platform to advance grounded visual reasoning in embodied AI systems.

Abstract

Pointing serves as a fundamental and intuitive mechanism for grounding language within visual contexts, with applications spanning robotics, assistive technologies, and interactive AI systems. While recent multimodal models have started to support pointing capabilities, existing benchmarks typically focus only on referential object localization tasks. We introduce PointArena, a comprehensive platform for evaluating multimodal pointing across diverse reasoning scenarios. PointArena comprises three components: (1) Point-Bench, a curated dataset containing approximately 1,000 pointing tasks across five reasoning categories; (2) Point-Battle, an interactive, web-based arena facilitating blind, pairwise model comparisons, which has already gathered over 4,500 anonymized votes; and (3) Point-Act, a real-world robotic manipulation system allowing users to directly evaluate multimodal model pointing capabilities in practical settings. We conducted extensive evaluations of both state-of-the-art open-source and proprietary multimodal models. Results indicate that Molmo-72B consistently outperforms other models, though proprietary models increasingly demonstrate comparable performance. Additionally, we find that supervised training specifically targeting pointing tasks significantly enhances model performance. Across our multi-stage evaluation pipeline, we also observe strong correlations, underscoring the critical role of precise pointing capabilities in enabling multimodal models to effectively bridge abstract reasoning with concrete, real-world actions. Project page: https://pointarena.github.io/

Figures (7)

• Figure 1: Overview of PointArena. PointArena consists of three components: Point-Bench, a curated dataset for evaluating grounded pointing across five reasoning types; Point-Battle, a live platform for blind, pairwise model comparisons with user voting; and Point-Act, real-world task involving manipulation via pointing-based language commands.
• Figure 2: Overview of the five Point-Bench categories and the annotation UI. Point-Bench includes 982 image-query pairs grouped into five categories: Spatial (positional references), Affordance (functional part identification), Counting (attribute-based grouping), Steerable (relative pointing), and Reasoning (open-ended visual inference). Each example shows a representative query and the corresponding target. On the right, we show the Gradio-based annotation interface used to collect and refine segmentation masks. Initial masks are generated using SAM and refined by annotators, followed by manual verification.
• Figure 3: Success rates of MLLMs on Point-Bench across six task categories: Spatial, Affordance, Counting, Steerable, Reasoning, and Average. Each bar represents the mean success rate (%) for a given model, with error bars indicating standard deviation across three evaluation runs. The "Human" bar serves as an upper-bound reference. The results demonstrate substantial performance disparities, with top models (e.g., GPT-4o, Gemini-2.5-Pro, Molmo-72B) achieving near-human accuracy in select categories, while others (e.g., LLaVA, Grok, and Claude) consistently underperform.
• Figure 4: Qualitative predictions across Point-Bench categories. Example model predictions are shown for each of the five Point-Bench categories: Spatial, Affordance, Counting, Steerable, and Reasoning. Each colored dot corresponds to a prediction from a different MLLM, labeled by model name in the legend. These examples highlight the diversity of pointing behaviors and the variation in performance across models.
• Figure 5: Insights drawn from Point-Battle and Point-Bench. (a) This figure shows Point-Bench performance (%) of MLLMs over time, grouped by model family. A sharp performance increase is observed in models released after the PixMo dataset or RoboPoint (dashed line, October 2024 or December 2024). Notably, GPT4.1 improves by 21.1 percentage points over GPT-4-Turbo, and Gemini-2.0-Flash improves by 45.9 points over Gemini-1.5-Flash. These trends suggest that newer proprietary models may incorporate pointing supervision, potentially derived from or inspired by RoboPoint or PixMo. (b) Linear regression of the five models common to Point-Battle and Point-Bench reveals a strong correlation ($R^{2}=0.85$), confirming close agreement between the two evaluation frameworks. (c) Performance of open-source models as a function of parameter count. While there is a slight upward trend, the performance gains with increasing model size are marginal, suggesting diminishing returns and limited sensitivity to scale within this range.