CapArena: Benchmarking and Analyzing Detailed Image Captioning in the LLM Era

TL;DR

The paper addresses the challenge of evaluating detailed image captions in the era of large language models. It introduces CapArena, a large-scale, human-centered benchmark with pairwise caption battles to rank VLMs and humans on detailed captions, revealing GPT-4o's near-human performance and persistent gaps for open-source models. It then analyzes captioning metrics and demonstrates that VLM-as-a-Judge yields the best alignment with human preferences, uncovering biases in traditional metrics. To enable scalable evaluation, the authors release CapArena-Auto, a 600-sample automated benchmark that achieves 94.3% correlation with human rankings at a cost of about $4 per test. Together, these contributions provide a reliable, scalable framework for advancing detailed captioning in vision–language systems.

Abstract

Image captioning has been a longstanding challenge in vision-language research. With the rise of LLMs, modern Vision-Language Models (VLMs) generate detailed and comprehensive image descriptions. However, benchmarking the quality of such captions remains unresolved. This paper addresses two key questions: (1) How well do current VLMs actually perform on image captioning, particularly compared to humans? We built CapArena, a platform with over 6000 pairwise caption battles and high-quality human preference votes. Our arena-style evaluation marks a milestone, showing that leading models like GPT-4o achieve or even surpass human performance, while most open-source models lag behind. (2) Can automated metrics reliably assess detailed caption quality? Using human annotations from CapArena, we evaluate traditional and recent captioning metrics, as well as VLM-as-a-Judge. Our analysis reveals that while some metrics (e.g., METEOR) show decent caption-level agreement with humans, their systematic biases lead to inconsistencies in model ranking. In contrast, VLM-as-a-Judge demonstrates robust discernment at both the caption and model levels. Building on these insights, we release CapArena-Auto, an accurate and efficient automated benchmark for detailed captioning, achieving 94.3% correlation with human rankings at just $4 per test. Data and resources will be open-sourced at https://caparena.github.io.

Figures (8)

• Figure 1: Model rankings from CapArena in detailed captioning. Top models are comparable to humans, while most open-source models lag behind.
• Figure 2: Caption length distribution of different VLMs.
• Figure 3: Battle counts and win rate matrix between models in CapArena.
• Figure 4: Correlation between vision-language benchmark scores and CapArena ranking. Models that perform well on general benchmarks do not necessarily excel in captioning. The size of each point represents the model size.
• Figure 5: Metrics exhibit systematic biases—overestimating (positive values) or underestimating (negative values) certain models. Color saturation represents bias magnitude. Different metrics favor different models. GPT-4o exhibits lower biases (lighter overall colors), contributing to higher model-level agreement observed in \ref{['tab:captioning_metrics']}.