PolySmart and VIREO @ TRECVid 2024 Ad-hoc Video Search

TL;DR

This work tackles ad-hoc video search with open vocabulary by applying generation-augmented retrieval to AVS queries. It leverages three generation modalities—Text2Text, Text2Image, and Image2Text—to replace out-of-vocabulary terms and encode spatial/logical constraints, fusing four rank lists (original plus generated variants). The automatic runs show that combining original and generated queries improves performance on TV24, with Text2Text contributing notable gains and Text2Image offering broader semantic shifts, while manual GPT-4 rephrasings yield limited improvements. The results demonstrate the potential of cross-modal generation to mitigate OOV issues in TRECVid AVS, though further improvements in image-to-video retrieval and query attention are needed for robust gains.

Abstract

This year, we explore generation-augmented retrieval for the TRECVid AVS task. Specifically, the understanding of textual query is enhanced by three generations, including Text2Text, Text2Image, and Image2Text, to address the out-of-vocabulary problem. Using different combinations of them and the rank list retrieved by the original query, we submitted four automatic runs. For manual runs, we use a large language model (LLM) (i.e., GPT4) to rephrase test queries based on the concept bank of the search engine, and we manually check again to ensure all the concepts used in the rephrased queries are in the bank. The result shows that the fusion of the original and generated queries outperforms the original query on TV24 query sets. The generated queries retrieve different rank lists from the original query.

Figures (5)

• Figure 1: The overview of the generation-augmented retrieval plug-in module.
• Figure 2: Example of an improved query by the generation.
• Figure 3: Example of an improved query by the generation.
• Figure 4: Example of an improved query by the generation.
• Figure 5: Example of an improved query by the generation.