LocateBench: Evaluating the Locating Ability of Vision Language Models

TL;DR

This work proposes LocateBench, a high-quality benchmark dedicated to evaluating the ability to locate an object in an image according to natural language instructions, and measures the accuracy of several large vision language models.

Abstract

The ability to locate an object in an image according to natural language instructions is crucial for many real-world applications. In this work we propose LocateBench, a high-quality benchmark dedicated to evaluating this ability. We experiment with multiple prompting approaches, and measure the accuracy of several large vision language models. We find that even the accuracy of the strongest model, GPT-4o, lags behind human accuracy by more than 10%.

Figures (5)

• Figure 1: Some examples from LocateBench. Questions in our dataset can be categorized into fine-grained descriptions (\ref{['fig:ex-fine-grained']}), relative size (\ref{['fig:ex-size']}), counting (\ref{['fig:ex-counting']}) or relative location (\ref{['fig:ex-rel']}).
• Figure 2: Less ideal examples in Pointing QA (§\ref{['sec:pointing-qa']}).
• Figure 3: The Venn diagrams of the errors made by the three VLMs with different prompts.
• Figure 4: The Venn diagrams of the errors made by the three VLMs with different prompts.
• Figure 5: Hard examples that all models got wrong in the multi-choice by alphabet letters (ABCD) setting.