Modeling Image-Caption Rating from Comparative Judgments

TL;DR

This work tackles the subjectivity and cost of rating image-caption alignment by proposing a comparative-judgment framework. It compares a regression model trained on absolute ratings with a comparative learning model trained on pairwise preferences, using a dual-encoder architecture with ResNet-50 for images and MiniLM for captions on the VICR dataset. Although the regression model achieves higher absolute correlation, the comparative model is data-efficient and scales toward the baseline with more comparisons, while also delivering strong performance on same-image caption ranking and substantial gains in annotation efficiency. A small human study confirms that comparative annotations are faster and yield higher inter-rater agreement, underscoring the practical benefits for large-scale subjective caption evaluation.

Abstract

Rating the accuracy of captions in describing images is time-consuming and subjective for humans. In contrast, it is often easier for people to compare two captions and decide which one better matches a given image. In this work, we propose a machine learning framework that models such comparative judgments instead of direct ratings. The model can then be applied to rank unseen image-caption pairs in the same way as a regression model trained on direct ratings. Using the VICR dataset, we extract visual features with ResNet-50 and text features with MiniLM, then train both a regression model and a comparative learning model. While the regression model achieves better performance (Pearson's $ρ$: 0.7609 and Spearman's $r_s$: 0.7089), the comparative learning model steadily improves with more data and approaches the regression baseline. In addition, a small-scale human evaluation study comparing absolute rating, pairwise comparison, and same-image comparison shows that comparative annotation yields faster results and has greater agreement among human annotators. These results suggest that comparative learning can effectively model human preferences while significantly reducing the cost of human annotations.

Figures (6)

• Figure 1: Regression framework showing preprocessing, multimodal fusion, and model training path.
• Figure 3: Task 1: Absolute rating of each caption on a 1–5 scale. Participants judged how well the caption described the corresponding image.
• Figure 4: Task 2: Pairwise comparison across different images. Participants selected which image-caption pair better matches each other.
• Figure 5: Task 3: Pairwise comparison of captions describing the same image. Participants selected which caption better matches the given image.
• Figure 6: Pearson ($\rho$) and Spearman ($r_s$) correlations improve with the number of sampled pairwise comparisons per caption ($N$), approaching the regression baseline.