Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

LLM-Free Image Captioning Evaluation in Reference-Flexible Settings

Shinnosuke Hirano, Yuiga Wada, Kazuki Matsuda, Seitaro Otsuki, Komei Sugiura

TL;DR

Pearl introduces a fast, LLM-free automatic evaluation metric for image captioning that works in both reference-based and reference-free settings. The method jointly learns image-caption and caption-caption similarities via a late-fusion architecture comprising Img-GEM and Ref-GEM, powered by Adaptive RUSE-type Similarity Mechanism and Vector Similarity Scoring with multiple encoders. A key contribution is Spica, a large human-annotated dataset that enables robust training and evaluation. Empirical results show Pearl achieving state-of-the-art performance among LLM-free metrics, strong human correlation, and competitive inference times, with ablations validating the value of ARSM, Stella/CLIP, and Spica-based training.

Abstract

We focus on the automatic evaluation of image captions in both reference-based and reference-free settings. Existing metrics based on large language models (LLMs) favor their own generations; therefore, the neutrality is in question. Most LLM-free metrics do not suffer from such an issue, whereas they do not always demonstrate high performance. To address these issues, we propose Pearl, an LLM-free supervised metric for image captioning, which is applicable to both reference-based and reference-free settings. We introduce a novel mechanism that learns the representations of image--caption and caption--caption similarities. Furthermore, we construct a human-annotated dataset for image captioning metrics, that comprises approximately 333k human judgments collected from 2,360 annotators across over 75k images. Pearl outperformed other existing LLM-free metrics on the Composite, Flickr8K-Expert, Flickr8K-CF, Nebula, and FOIL datasets in both reference-based and reference-free settings. Our project page is available at https://pearl.kinsta.page/.

LLM-Free Image Captioning Evaluation in Reference-Flexible Settings

TL;DR

Pearl introduces a fast, LLM-free automatic evaluation metric for image captioning that works in both reference-based and reference-free settings. The method jointly learns image-caption and caption-caption similarities via a late-fusion architecture comprising Img-GEM and Ref-GEM, powered by Adaptive RUSE-type Similarity Mechanism and Vector Similarity Scoring with multiple encoders. A key contribution is Spica, a large human-annotated dataset that enables robust training and evaluation. Empirical results show Pearl achieving state-of-the-art performance among LLM-free metrics, strong human correlation, and competitive inference times, with ablations validating the value of ARSM, Stella/CLIP, and Spica-based training.

Abstract

We focus on the automatic evaluation of image captions in both reference-based and reference-free settings. Existing metrics based on large language models (LLMs) favor their own generations; therefore, the neutrality is in question. Most LLM-free metrics do not suffer from such an issue, whereas they do not always demonstrate high performance. To address these issues, we propose Pearl, an LLM-free supervised metric for image captioning, which is applicable to both reference-based and reference-free settings. We introduce a novel mechanism that learns the representations of image--caption and caption--caption similarities. Furthermore, we construct a human-annotated dataset for image captioning metrics, that comprises approximately 333k human judgments collected from 2,360 annotators across over 75k images. Pearl outperformed other existing LLM-free metrics on the Composite, Flickr8K-Expert, Flickr8K-CF, Nebula, and FOIL datasets in both reference-based and reference-free settings. Our project page is available at https://pearl.kinsta.page/.
Paper Structure (50 sections, 2 equations, 3 figures, 9 tables)

This paper contains 50 sections, 2 equations, 3 figures, 9 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. LLM-free Metrics
  4. Reference-based metrics.
  5. Reference-free metrics.
  6. LLM-based Metrics
  7. Methodology
  8. Why do we need LLM-free metrics?
  9. Architecture Overview
  10. Adaptive RUSE-type Similarity Mechanism
  11. DiffNet.
  12. HadamardNet.
  13. Img-GEM and Ref-GEM
  14. Vector Similarity Scoring Submodule
  15. Spica Dataset
  16. ...and 35 more sections

Figures (3)

  • Figure 1: Pearl is an LLM-free automatic evaluation metric for image captioning. Pearl works significantly faster than slow LLM-based metrics; thus it is suitable for fast development cycles of practical image captioning models. Moreover, Pearl is the first LLM-free supervised metric that can handle both reference-based and reference-free evaluation with a single supervised model.
  • Figure 2: Overview of the proposed metric, Pearl. Our proposed metric consists of the Img-GEM and multiple Ref-GEMs. The Img-GEM computes a score for a candidate caption based on the associated image, whereas each Ref-GEM calculates a score based on a reference. In reference-based setting, Pearl computes the scores for candidate caption based on either the image or the references, and then fuses them into the final prediction score. Conversely, in the reference-free setting, the final prediction score is computed based solely on the image.
  • Figure 3: Qualitative results on the Nebula dataset. Cases (a) and (b) show successful cases in the reference-based setting, while Case (c) highlights successful sample in the reference-free setting. Case (d) shows a failure case in the reference-free setting.