Technical Report of NICE Challenge at CVPR 2024: Caption Re-ranking Evaluation Using Ensembled CLIP and Consensus Scores

TL;DR

The paper tackles zero-shot image caption re-ranking by introducing ECO, a framework that jointly optimizes semantic alignment and caption essentialness. ECO computes an Ensembled CLIP score across multiple CLIP models and a Consensus score based on cross-caption essential expressions, then fuses them via $S_{\text{comb}}=\lambda_1 S'_{\text{ensemble}}+\lambda_2 S'_{\text{consensus}}$ with a empirically favored ratio of $3.52:1$. It incorporates two filtering steps (Bad Format and ITM) to improve the caption pool and uses Short Caption Selection to break ties in favor of brevity, all in a zero-shot setting. Experiments on the NICE 2024 Caption Re-ranking task show ECO achieves top performance across ROUGE-L and BLEU, and strong ranks in CIDEr, SPICE, and METEOR, demonstrating versatility across metrics without model fine-tuning. The authors release code at GitHub, highlighting ECO as a practical, training-free approach to robust caption re-ranking in diverse evaluation settings.

Abstract

This report presents the ECO (Ensembled Clip score and cOnsensus score) pipeline from team DSBA LAB, which is a new framework used to evaluate and rank captions for a given image. ECO selects the most accurate caption describing image. It is made possible by combining an Ensembled CLIP score, which considers the semantic alignment between the image and captions, with a Consensus score that accounts for the essentialness of the captions. Using this framework, we achieved notable success in the CVPR 2024 Workshop Challenge on Caption Re-ranking Evaluation at the New Frontiers for Zero-Shot Image Captioning Evaluation (NICE). Specifically, we secured third place based on the CIDEr metric, second in both the SPICE and METEOR metrics, and first in the ROUGE-L and all BLEU Score metrics. The code and configuration for the ECO framework are available at https://github.com/DSBA-Lab/ECO .

Figures (6)

• Figure 1: Overview of ECO (Ensembled Clip score and cOnsensus score) framework utilizes both the Ensembled CLIP score and Consensus score to select the ideal caption.
• Figure 2: The Short Cap Selection process: If the final scores of the top 2 captions differ by less than the threshold $\mathbf{\theta}$, a caption with fewer words is chosen.
• Figure 3: Comparison of the Ensembled CLIP score and Consensus score distributions
• Figure 4: Comparison of the number of words in captions selected by the Ensembled CLIP score versus the Consensus score.
• Figure 5: The relative rank of the selected caption within the candidate caption pool in terms of the number of words.