Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

VCRScore: Image captioning metric based on V\&L Transformers, CLIP, and precision-recall

Guillermo Ruiz, Tania Ramírez, Daniela Moctezuma

TL;DR

This work generated a human-labeled dataset to assess to which degree the captions correlate with the image's content, and proposes a new evaluation metric for the image captioning problem.

Abstract

Image captioning has become an essential Vision & Language research task. It is about predicting the most accurate caption given a specific image or video. The research community has achieved impressive results by continuously proposing new models and approaches to improve the overall model's performance. Nevertheless, despite increasing proposals, the performance metrics used to measure their advances have remained practically untouched through the years. A probe of that, nowadays metrics like BLEU, METEOR, CIDEr, and ROUGE are still very used, aside from more sophisticated metrics such as BertScore and ClipScore. Hence, it is essential to adjust how are measure the advances, limitations, and scopes of the new image captioning proposals, as well as to adapt new metrics to these new advanced image captioning approaches. This work proposes a new evaluation metric for the image captioning problem. To do that, first, it was generated a human-labeled dataset to assess to which degree the captions correlate with the image's content. Taking these human scores as ground truth, we propose a new metric, and compare it with several well-known metrics, from classical to newer ones. Outperformed results were also found, and interesting insights were presented and discussed.

VCRScore: Image captioning metric based on V\&L Transformers, CLIP, and precision-recall

TL;DR

This work generated a human-labeled dataset to assess to which degree the captions correlate with the image's content, and proposes a new evaluation metric for the image captioning problem.

Abstract

Image captioning has become an essential Vision & Language research task. It is about predicting the most accurate caption given a specific image or video. The research community has achieved impressive results by continuously proposing new models and approaches to improve the overall model's performance. Nevertheless, despite increasing proposals, the performance metrics used to measure their advances have remained practically untouched through the years. A probe of that, nowadays metrics like BLEU, METEOR, CIDEr, and ROUGE are still very used, aside from more sophisticated metrics such as BertScore and ClipScore. Hence, it is essential to adjust how are measure the advances, limitations, and scopes of the new image captioning proposals, as well as to adapt new metrics to these new advanced image captioning approaches. This work proposes a new evaluation metric for the image captioning problem. To do that, first, it was generated a human-labeled dataset to assess to which degree the captions correlate with the image's content. Taking these human scores as ground truth, we propose a new metric, and compare it with several well-known metrics, from classical to newer ones. Outperformed results were also found, and interesting insights were presented and discussed.
Paper Structure (20 sections, 7 equations, 10 figures, 11 tables)

This paper contains 20 sections, 7 equations, 10 figures, 11 tables.

Table of Contents

  1. Introduction
  2. Problem statement
  3. Contribution
  4. Dataset generation
  5. Tagging process
  6. Taggers performance analysis
  7. Models's performance
  8. Proposal Metric
  9. Metrics used
  10. State-of-the-art metrics
  11. Model's embeddings as metrics
  12. Precision and Recall calculation
  13. Gradient Boosting Regressor
  14. Spearman Correlation
  15. Experiments and analysis
  16. ...and 5 more sections

Figures (10)

  • Figure 1: Screenshot of the simple tagging designed tool
  • Figure 2: Overall score for each model, as well as the Reference of the MS-COCO dataset
  • Figure 3: Scores assigned to each model's prediction by all the taggers in the two stages
  • Figure 4: Examples of captions in the different datasets. (a) is for VICR, (b) is Flickr8k-Expert, (c) is Flickr8k-CF, and (d) Composite. The normalized score is in parentheses.
  • Figure 5: HeatMap of the correlation results of all the metrics evaluated
  • ...and 5 more figures