VELA: An LLM-Hybrid-as-a-Judge Approach for Evaluating Long Image Captions

TL;DR

This work targets automatic evaluation of long image captions produced by multimodal LLMs, where existing metrics struggle to reflect human judgments. It introduces VELA, a supervised metric built on a novel LLM-Hybrid-as-a-Judge framework with two branches (R2C-LLM and I2C-Align) that enables fast, multi-perspective evaluation across Descriptiveness, Relevance, and Fluency. The LongCap-Arena benchmark is proposed to train and evaluate metrics for long captions, comprising 7,805 images, long references, long candidates, and 32,246 human judgments, facilitating robust alignment with human judgments. Empirically, VELA outperforms baselines and even reaches superhuman performance on LongCap-Arena, while maintaining substantially faster inference than autoregressive LLM-based evaluators, as evidenced by Kendall's $\tau$ statistics across Desc., Rel., and Fluency.

Abstract

In this study, we focus on the automatic evaluation of long and detailed image captions generated by multimodal Large Language Models (MLLMs). Most existing automatic evaluation metrics for image captioning are primarily designed for short captions and are not suitable for evaluating long captions. Moreover, recent LLM-as-a-Judge approaches suffer from slow inference due to their reliance on autoregressive inference and early fusion of visual information. To address these limitations, we propose VELA, an automatic evaluation metric for long captions developed within a novel LLM-Hybrid-as-a-Judge framework. Furthermore, we propose LongCap-Arena, a benchmark specifically designed for evaluating metrics for long captions. This benchmark comprises 7,805 images, the corresponding human-provided long reference captions and long candidate captions, and 32,246 human judgments from three distinct perspectives: Descriptiveness, Relevance, and Fluency. We demonstrated that VELA outperformed existing metrics and achieved superhuman performance on LongCap-Arena.

Figures (5)

• Figure 1: Overview of Vela, which evaluates long image captions from three perspectives: Descriptiveness, Relevance, and Fluency. Vela employs an LLM-Hybrid-as-a-Judge framework, which enables both computational efficiency and high alignment with human judgments.
• Figure 2: Example of automatic evaluation for long captions. In this task, automatic evaluation metrics assess a candidate based on the given image and human-provided long references across three perspectives: Descriptiveness, Relevance, and Fluency. The evaluation scores should align with human judgments.
• Figure 3: Architecture of Vela. The image, long candidate, and human-provided long references are processed by our metric through two branches: R2C-LLM and I2C-Align. The R2C-LLM branch leverages an LLM to capture the linguistic relationship between the candidate and references, whereas the I2C-Align branch uses Long-CLIP to compute the similarity between the candidate and image.
• Figure 4: Qualitative results on LongCap-Arena. The left and middle subfigures illustrate successful cases, while the right subfigure shows a failure case. Each subfigure consists of $\bm{x}_{\text{img}}$, $\bm{x}_{\text{ref}}$ ("Reference"), $\bm{x}_{\text{cand}}$ ("Candidate"), and human judgments $\bm{y}$ along with automatic evaluation scores $\hat{\bm{y}}$ ("Human judgments & automatic evaluation"). Values in green and red indicate scores that are closely aligned and misaligned with human judgments, respectively.
• Figure 5: Annotation interface for Desc. The left subfigure shows the normal image, and the right subfigure presents its segmented version generated using SAM sam. These object masks were shown to annotators as visual cues, helping them determine the level of detail required for their evaluation.