Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

PREGEN: Uncovering Latent Thoughts in Composed Video Retrieval

Gabriele Serussi, David Vainshtein, Jonathan Kouchly, Dotan Di Castro, Chaim Baskin

TL;DR

This work tackles Composed Video Retrieval (CoVR) by addressing the limitations of finetuning large Vision-Language Models (VLMs) and caption-generation bottlenecks. It introduces PREGEN, a framework that freezes a VLM and aggregates hidden states from all layers through a lightweight Transformer encoder to produce compact, semantically rich embeddings for retrieval, trained with a symmetric InfoNCE loss and a novel source-based hard negative mining strategy. PREGEN achieves state-of-the-art results on WebVid-CoVR and FineCVR, demonstrates backbone-agnostic robustness across multiple VLM variants, and generalizes well to more complex textual modifications without retraining. The approach highlights the value of multi-layer VLM representations for multimodal retrieval, offering a scalable and efficient solution with broad applicability beyond CoVR.

Abstract

Composed Video Retrieval (CoVR) aims to retrieve a video based on a query video and a modifying text. Current CoVR methods fail to fully exploit modern Vision-Language Models (VLMs), either using outdated architectures or requiring computationally expensive fine-tuning and slow caption generation. We introduce PREGEN (PRE GENeration extraction), an efficient and powerful CoVR framework that overcomes these limitations. Our approach uniquely pairs a frozen, pre-trained VLM with a lightweight encoding model, eliminating the need for any VLM fine-tuning. We feed the query video and modifying text into the VLM and extract the hidden state of the final token from each layer. A simple encoder is then trained on these pooled representations, creating a semantically rich and compact embedding for retrieval. PREGEN significantly advances the state of the art, surpassing all prior methods on standard CoVR benchmarks with substantial gains in Recall@1 of +27.23 and +69.59. Our method demonstrates robustness across different VLM backbones and exhibits strong zero-shot generalization to more complex textual modifications, highlighting its effectiveness and semantic capabilities.

PREGEN: Uncovering Latent Thoughts in Composed Video Retrieval

TL;DR

This work tackles Composed Video Retrieval (CoVR) by addressing the limitations of finetuning large Vision-Language Models (VLMs) and caption-generation bottlenecks. It introduces PREGEN, a framework that freezes a VLM and aggregates hidden states from all layers through a lightweight Transformer encoder to produce compact, semantically rich embeddings for retrieval, trained with a symmetric InfoNCE loss and a novel source-based hard negative mining strategy. PREGEN achieves state-of-the-art results on WebVid-CoVR and FineCVR, demonstrates backbone-agnostic robustness across multiple VLM variants, and generalizes well to more complex textual modifications without retraining. The approach highlights the value of multi-layer VLM representations for multimodal retrieval, offering a scalable and efficient solution with broad applicability beyond CoVR.

Abstract

Composed Video Retrieval (CoVR) aims to retrieve a video based on a query video and a modifying text. Current CoVR methods fail to fully exploit modern Vision-Language Models (VLMs), either using outdated architectures or requiring computationally expensive fine-tuning and slow caption generation. We introduce PREGEN (PRE GENeration extraction), an efficient and powerful CoVR framework that overcomes these limitations. Our approach uniquely pairs a frozen, pre-trained VLM with a lightweight encoding model, eliminating the need for any VLM fine-tuning. We feed the query video and modifying text into the VLM and extract the hidden state of the final token from each layer. A simple encoder is then trained on these pooled representations, creating a semantically rich and compact embedding for retrieval. PREGEN significantly advances the state of the art, surpassing all prior methods on standard CoVR benchmarks with substantial gains in Recall@1 of +27.23 and +69.59. Our method demonstrates robustness across different VLM backbones and exhibits strong zero-shot generalization to more complex textual modifications, highlighting its effectiveness and semantic capabilities.
Paper Structure (34 sections, 5 equations, 4 figures, 8 tables)

This paper contains 34 sections, 5 equations, 4 figures, 8 tables.

Table of Contents

  1. Introduction
  2. Our contribution.
  3. Related Work
  4. Composed Image Retrieval (CIR).
  5. Composed Video Retrieval (CoVR).
  6. Vision Language Models (VLMs).
  7. Universal multimodal retrievers and LMM-based retrieval.
  8. Method
  9. Problem
  10. Model Architecture
  11. Training
  12. Contrastive learning.
  13. Hard negative mining using source-based batching.
  14. Experiments
  15. The effect of using all layers
  16. ...and 19 more sections

Figures (4)

  • Figure 2: PREGEN extracts the hidden state of the last token at every VLM layer. These vectors are position encoded and fed into a Transformer Encoder with a [CLS] token. The [CLS] token output is then projected through an MLP to produce the final embedding.
  • Figure : (a) Other VLM-based embedding methods
  • Figure : (a) Other VLM-based embedding methods
  • Figure : (b) PREGEN (Ours)