Expertized Caption Auto-Enhancement for Video-Text Retrieval

TL;DR

The paper tackles information mismatch in video-text retrieval by shifting augmentation to the video side through automatic caption enhancement. It introduces ExCae, a data-driven framework combining Caption Self-improvement (CSI) and Expertized Caption Selection (ECS) to generate and selectively fuse multi-perspective video-derived captions for tighter cross-modal alignment. Key contributions include an iterative prompt-engineered caption generation process, a mixture-of-experts module for adaptive caption-expression matching, and comprehensive evaluations showing state-of-the-art or competitive results on MSR-VTT, MSVD, and DiDeMo without extra data. The approach reduces reliance on heavy data collection and hand-crafted prompts, offering a plug-in, scalable solution with strong generalization and efficiency advantages for video-text understanding.

Abstract

Video-text retrieval has been stuck in the information mismatch caused by personalized and inadequate textual descriptions of videos. The substantial information gap between the two modalities hinders an effective cross-modal representation alignment, resulting in ambiguous retrieval results. Although text rewriting methods have been proposed to broaden text expressions, the modality gap remains significant, as the text representation space is hardly expanded with insufficient semantic enrichment.Instead, this paper turns to enhancing visual presentation, bridging video expression closer to textual representation via caption generation and thereby facilitating video-text matching.While multimodal large language models (mLLM) have shown a powerful capability to convert video content into text, carefully crafted prompts are essential to ensure the reasonableness and completeness of the generated captions. Therefore, this paper proposes an automatic caption enhancement method that improves expression quality and mitigates empiricism in augmented captions through self-learning.Additionally, an expertized caption selection mechanism is designed and introduced to customize augmented captions for each video, further exploring the utilization potential of caption augmentation.Our method is entirely data-driven, which not only dispenses with heavy data collection and computation workload but also improves self-adaptability by circumventing lexicon dependence and introducing personalized matching. The superiority of our method is validated by state-of-the-art results on various benchmarks, specifically achieving Top-1 recall accuracy of 68.5% on MSR-VTT, 68.1% on MSVD, and 62.0% on DiDeMo. Our code is publicly available at https://github.com/CaryXiang/ECA4VTR.

Figures (8)

• Figure 1: Comprehension gap between videos and texts
• Figure 2: Comparison of our method to the traditional
• Figure 3: Illustration of the CSI module
• Figure 4: Examples of text-to-video retrieval results on MSR-VTT dataset. (The ground truths are marked in a red box.)
• Figure 5: Examples of video-to-text retrieval results on MSR-VTT dataset. (The ground truths are marked in a red box.)