Beyond Coarse-Grained Matching in Video-Text Retrieval

TL;DR

A new approach for fine-grained evaluation that can be applied to existing datasets by automatically generating hard negative test captions with subtle single-word variations across nouns, verbs, adjectives, adverbs, and prepositions is introduced.

Abstract

Video-text retrieval has seen significant advancements, yet the ability of models to discern subtle differences in captions still requires verification. In this paper, we introduce a new approach for fine-grained evaluation. Our approach can be applied to existing datasets by automatically generating hard negative test captions with subtle single-word variations across nouns, verbs, adjectives, adverbs, and prepositions. We perform comprehensive experiments using four state-of-the-art models across two standard benchmarks (MSR-VTT and VATEX) and two specially curated datasets enriched with detailed descriptions (VLN-UVO and VLN-OOPS), resulting in a number of novel insights: 1) our analyses show that the current evaluation benchmarks fall short in detecting a model's ability to perceive subtle single-word differences, 2) our fine-grained evaluation highlights the difficulty models face in distinguishing such subtle variations. To enhance fine-grained understanding, we propose a new baseline that can be easily combined with current methods. Experiments on our fine-grained evaluations demonstrate that this approach enhances a model's ability to understand fine-grained differences.

Figures (8)

• Figure 1: Coarse vs. Fine-Grained Video Retrieval.
• Figure 2: Qualitative Analysis on MSR-VTT & VATEX. We show ground-truth video-text pairs alongside the 5 most similar captions to the ground-truth according to CLIP. We see that captions are coarse-grained, i.e. there are multiple difference conceptual differences between captions which mostly relate to the nouns present.
• Figure 3: Analysis of Sentence Similarity. Few captions differ by one part-of-speech, particularly outside of nouns.
• Figure 4: Examples of Hard Negative Sentences in our evaluation. By creating subtle single-word variations of the original sentence we test the ability of video-text retrieval models to distinguish fine-grained variations for different parts-of-speech.
• Figure 5: Correlation Between Coarse & Fine-Grained Performance. There is a weak correlation between performance on coarse and fine-grained metrics, however good coarse-grained performance does not indicate good fine-grained performance.