Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

VIRTUE: Versatile Video Retrieval Through Unified Embeddings

Shaunak Halbe, Bhagyashree Puranik, Jayakrishnan Unnikrishnan, Kushan Thakkar, Vimal Bhat, Toufiq Parag

TL;DR

VIRTUE presents a unified, retrieval-centric framework that leverages contrastively aligned unified embeddings from a multimodal LLM backbone to perform corpus-level video retrieval with reranking, zero-shot composed video retrieval, and zero-shot moment localization within a single architecture. The method uses a two-stage contrastive training pipeline (image-text then video-text) with LoRA adapters to obtain VIRTUE-Embed, and a separate VIRTUE-Ranker to refine top candidates, all without task-specific fine-tuning. Empirically, VIRTUE achieves strong zero-shot performance on standard benchmarks, surpasses many MLLM-based baselines, and attains state-of-the-art results on zero-shot composed retrieval on CoVR, while remaining competitive with specialized models that rely on much larger data. The results demonstrate the practicality and scalability of retrieval-oriented contrastive adaptation for unified video understanding and pave the way for flexible, multi-task video retrieval engines.

Abstract

Modern video retrieval systems are expected to handle diverse tasks ranging from corpus-level retrieval and fine-grained moment localization to flexible multimodal querying. Specialized architectures achieve strong retrieval performance by training modality-specific encoders on massive datasets, but they lack the ability to process composed multimodal queries. In contrast, multimodal LLM (MLLM)-based methods support rich multimodal search but their retrieval performance remains well below that of specialized systems. We present VIRTUE, an MLLM-based versatile video retrieval framework that integrates corpus and moment-level retrieval capabilities while accommodating composed multimodal queries within a single architecture. We use contrastive alignment of visual and textual embeddings generated using a shared MLLM backbone to facilitate efficient embedding-based candidate search. Our embedding model, trained efficiently using low-rank adaptation (LoRA) on 700K paired visual-text data samples, surpasses other MLLM-based methods on zero-shot video retrieval tasks. Additionally, we demonstrate that the same model can be adapted without further training to achieve competitive results on zero-shot moment retrieval, and state of the art results for zero-shot composed video retrieval. With additional training for reranking candidates identified in the embedding-based search, our model substantially outperforms existing MLLM-based retrieval systems and achieves retrieval performance comparable to state of the art specialized models which are trained on orders of magnitude larger data.

VIRTUE: Versatile Video Retrieval Through Unified Embeddings

TL;DR

VIRTUE presents a unified, retrieval-centric framework that leverages contrastively aligned unified embeddings from a multimodal LLM backbone to perform corpus-level video retrieval with reranking, zero-shot composed video retrieval, and zero-shot moment localization within a single architecture. The method uses a two-stage contrastive training pipeline (image-text then video-text) with LoRA adapters to obtain VIRTUE-Embed, and a separate VIRTUE-Ranker to refine top candidates, all without task-specific fine-tuning. Empirically, VIRTUE achieves strong zero-shot performance on standard benchmarks, surpasses many MLLM-based baselines, and attains state-of-the-art results on zero-shot composed retrieval on CoVR, while remaining competitive with specialized models that rely on much larger data. The results demonstrate the practicality and scalability of retrieval-oriented contrastive adaptation for unified video understanding and pave the way for flexible, multi-task video retrieval engines.

Abstract

Modern video retrieval systems are expected to handle diverse tasks ranging from corpus-level retrieval and fine-grained moment localization to flexible multimodal querying. Specialized architectures achieve strong retrieval performance by training modality-specific encoders on massive datasets, but they lack the ability to process composed multimodal queries. In contrast, multimodal LLM (MLLM)-based methods support rich multimodal search but their retrieval performance remains well below that of specialized systems. We present VIRTUE, an MLLM-based versatile video retrieval framework that integrates corpus and moment-level retrieval capabilities while accommodating composed multimodal queries within a single architecture. We use contrastive alignment of visual and textual embeddings generated using a shared MLLM backbone to facilitate efficient embedding-based candidate search. Our embedding model, trained efficiently using low-rank adaptation (LoRA) on 700K paired visual-text data samples, surpasses other MLLM-based methods on zero-shot video retrieval tasks. Additionally, we demonstrate that the same model can be adapted without further training to achieve competitive results on zero-shot moment retrieval, and state of the art results for zero-shot composed video retrieval. With additional training for reranking candidates identified in the embedding-based search, our model substantially outperforms existing MLLM-based retrieval systems and achieves retrieval performance comparable to state of the art specialized models which are trained on orders of magnitude larger data.
Paper Structure (37 sections, 2 equations, 3 figures, 9 tables)

This paper contains 37 sections, 2 equations, 3 figures, 9 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Corpus-level video retrieval
  4. Multimodal Large Language Models
  5. MLLMs as unified embedding models
  6. MLLMs for moment retrieval
  7. Method
  8. VIRTUE-Embed: Contrastive Visual Embedding Learning
  9. VIRTUE-Ranker: Candidate Refinement
  10. Zero-shot Moment Retrieval
  11. Zero-shot Composed Video Retrieval
  12. Experimental Analysis
  13. Text-to-Video Retrieval
  14. Training Details
  15. Evaluation
  16. ...and 22 more sections

Figures (3)

  • Figure 1: VIRTUE supports corpus-level retrieval with reranking, zero-shot composed video retrieval, and zero-shot moment localization within a single architecture. The tables (left) highlight that VIRTUE uniquely offers unified embeddings and versatile capabilities without relying on multi-task instruction tuning. $\diamond$ indicates models that, while architecturally capable of processing multimodal inputs, have not demonstrated composed video retrieval capability.
  • Figure 2: VIRTUE-Embed uses the final hidden state of the EOS token as an embedding anchor, and aligns visual content and text descriptions through contrastive learning.
  • Figure 3: VIRTUE-Ranker re-scores each query–video pair by feeding them jointly through the MLLM and projecting the EOS hidden state to a pointwise matching score.