ECCV Caption: Correcting False Negatives by Collecting Machine-and-Human-verified Image-Caption Associations for MS-COCO

TL;DR

The paper identifies widespread false negatives in image-caption ITM benchmarks caused by one-to-one positive pairs and proposes ECCV Caption, a corrected benchmark built via a machine-in-the-loop process using five diverse ITM models plus human verification. ECCV Caption substantially increases positives (about $3.6\times$ image-to-caption and $8.5\times$ caption-to-image) and introduces $mAP@R$ as a human-aligned ranking metric, demonstrated through a human study. Re-evaluating 25 VL models on ECCV Caption reveals rankings that diverge from traditional Recall-based benchmarks, highlighting model biases and the impact of annotation strategies; using multiple machine annotators mitigates these biases. The dataset and code enable fairer, more informative ITM evaluation and emphasize reporting both diversity and ranking quality rather than top-1 accuracy alone.

Abstract

Image-Text matching (ITM) is a common task for evaluating the quality of Vision and Language (VL) models. However, existing ITM benchmarks have a significant limitation. They have many missing correspondences, originating from the data construction process itself. For example, a caption is only matched with one image although the caption can be matched with other similar images and vice versa. To correct the massive false negatives, we construct the Extended COCO Validation (ECCV) Caption dataset by supplying the missing associations with machine and human annotators. We employ five state-of-the-art ITM models with diverse properties for our annotation process. Our dataset provides x3.6 positive image-to-caption associations and x8.5 caption-to-image associations compared to the original MS-COCO. We also propose to use an informative ranking-based metric mAP@R, rather than the popular Recall@K (R@K). We re-evaluate the existing 25 VL models on existing and proposed benchmarks. Our findings are that the existing benchmarks, such as COCO 1K R@K, COCO 5K R@K, CxC R@1 are highly correlated with each other, while the rankings change when we shift to the ECCV mAP@R. Lastly, we delve into the effect of the bias introduced by the choice of machine annotator. Source code and dataset are available at https://github.com/naver-ai/eccv-caption

Figures (15)

• Figure 1: Inherent multiplicity of correspondences in MS-COCO Caption. While any image-caption pair above makes sense (positive pair), only red and blue image-caption pairs are marked as positive in MS-COCO Caption.
• Figure 2: ECCV Caption examples. The given caption query: "A herd of zebras standing together in the field". Red: original positive. Green: annotated as "100% Yes". Blue: annotated as "Weak Yes". More examples are in \ref{['subsec:eccv_examples']}.
• Figure 3: Multiplicity in ECCV Caption. (a) The number of positive pairs in ECCV Caption. Dashed lines denote the number of the original COCO positives (1 image for each caption, and 5 captions for each image). ECCV Caption contains plenty of positive items per each modality. (b) PCME-predicted multiplicity against the number of positive captions for each image. There exists a positive correlation.
• Figure 4: Ranking correlation between different evaluation metrics. Ranking of methods is largely perserved between COCO and CxC Recall@1, while it is rarely preserved among COCO Recall@1, ECCV mAP@R and PMRP.
• Figure 5: Rankings of different VL models. Ranking of (a) PVSE models with diverse triplet mining strategies (b) contrastive methods (c) the best models are shown.