Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

DREAM: Improving Video-Text Retrieval Through Relevance-Based Augmentation Using Large Foundation Models

Yimu Wang, Shuai Yuan, Bo Xue, Xiangru Jian, Wei Pang, Mushi Wang, Ning Yu

TL;DR

The paper tackles the data quality bottleneck in video-text retrieval caused by ambiguous one-to-one annotations. It introduces dReAm, a relevance-based augmentation framework that combines simple self-augmentation, text paraphrasing with video stylization, and relevance-enriching data using large language models and visual generative models, built on an X-CLIP backbone with a symmetric contrastive loss. The approach yields state-of-the-art results on MSR-VTT, MSVD, and ActivityNet, with ablations illustrating the substantial impact of relevance-enhancing augmentations and paraphrase-based refinements. This work demonstrates that foundation-model-driven data augmentation can substantially improve cross-modal retrieval robustness and generalization, with potential implications for broader video-language understanding tasks.

Abstract

Recent progress in video-text retrieval has been driven largely by advancements in model architectures and training strategies. However, the representation learning capabilities of videotext retrieval models remain constrained by lowquality and limited training data annotations. To address this issue, we present a novel ViDeoText Retrieval Paradigm with RElevance-based AugMentation, namely DREAM, which enhances video and text data using large foundation models to learn more generalized features. Specifically, we first adopt a simple augmentation method, which generates self-similar data by randomly duplicating or dropping subwords and frames. In addition, inspired by the recent advancement in visual and language generative models, we propose a more robust augmentation method through textual paraphrasing and video stylization using large language models (LLMs) and visual generative models (VGMs). To further enrich video and text information, we propose a relevance-based augmentation method, where LLMs and VGMs generate and integrate new relevant information into the original data. Leveraging this enriched data, extensive experiments on several video-text retrieval benchmarks demonstrate the superiority of DREAM over existing methods.

DREAM: Improving Video-Text Retrieval Through Relevance-Based Augmentation Using Large Foundation Models

TL;DR

The paper tackles the data quality bottleneck in video-text retrieval caused by ambiguous one-to-one annotations. It introduces dReAm, a relevance-based augmentation framework that combines simple self-augmentation, text paraphrasing with video stylization, and relevance-enriching data using large language models and visual generative models, built on an X-CLIP backbone with a symmetric contrastive loss. The approach yields state-of-the-art results on MSR-VTT, MSVD, and ActivityNet, with ablations illustrating the substantial impact of relevance-enhancing augmentations and paraphrase-based refinements. This work demonstrates that foundation-model-driven data augmentation can substantially improve cross-modal retrieval robustness and generalization, with potential implications for broader video-language understanding tasks.

Abstract

Recent progress in video-text retrieval has been driven largely by advancements in model architectures and training strategies. However, the representation learning capabilities of videotext retrieval models remain constrained by lowquality and limited training data annotations. To address this issue, we present a novel ViDeoText Retrieval Paradigm with RElevance-based AugMentation, namely DREAM, which enhances video and text data using large foundation models to learn more generalized features. Specifically, we first adopt a simple augmentation method, which generates self-similar data by randomly duplicating or dropping subwords and frames. In addition, inspired by the recent advancement in visual and language generative models, we propose a more robust augmentation method through textual paraphrasing and video stylization using large language models (LLMs) and visual generative models (VGMs). To further enrich video and text information, we propose a relevance-based augmentation method, where LLMs and VGMs generate and integrate new relevant information into the original data. Leveraging this enriched data, extensive experiments on several video-text retrieval benchmarks demonstrate the superiority of DREAM over existing methods.
Paper Structure (23 sections, 1 equation, 4 figures, 13 tables)

This paper contains 23 sections, 1 equation, 4 figures, 13 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. Method
  4. Problem Definition
  5. dReAm
  6. Simple Augmentation (SA)
  7. Augmentation by Text Paraphrasing and Video Stylization (TPVS)
  8. Text paraphrasing
  9. Video sylization
  10. Augmentation by Relevance Enhancing (RE)
  11. Text relevance enhancing
  12. Video relevance enhancing
  13. Base Model and Training Objectives
  14. Experiments
  15. Quantitative Results
  16. ...and 8 more sections

Figures (4)

  • Figure 1: Existing video retrieval works focus on improving representation learning ability by learning from benchmarks that have many semantically similar data points, as shown in the top rows. It leads to vague annotations and associations between videos and texts, further hindering the representation learning ability of video-text retrieval models. To counteract this issue, in this study, we propose dReAm. Specifically, instead of learning from original noisy data, dReAm augments data with three proposed augmentation methods, i.e., simple augmentation, augmentation by text paraphrasing and video stylization ("Aug. 2 Parapharse" in the figure), and augmentation by relevance enhancing ("Aug. 3 Relevance Enhance" in the figure).
  • Figure 2: Qualitative examples of data generated by dReAm. "Aug. 2" and "Aug. 3" refer to augmentation by text paraphrasing and video stylization and augmentation by relevance enhancing.
  • Figure 3: Retrieval examples by the baseline and dReAm. "Rank x" means that the example is ranked at $x$. The numbers in blue represent the similarity to the query. Texts in blue and video frames surrounded by green lines are augmented data.
  • Figure 4: Qualitative examples of data generated by dReAm. "Aug. 2" and "Aug. 3" refer to augmentation by text paraphrasing and video stylization and augmentation by relevance enhancing.