Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

VCR: Video representation for Contextual Retrieval

Oron Nir, Idan Vidra, Avi Neeman, Barak Kinarti, Ariel Shamir

TL;DR

VCR proposes a multimodal, text-based video representation framework for contextual retrieval by fusing ASR, OCR, and frame-caption signals into latent semantic embeddings indexed offline and queried online via GPT-4–generated descriptions. The approach includes a supervised multilabel classifier and a learning-free GPT-based encoder, both yielding a joint semantic space, and introduces the Topics-Map visualization to support ontology-driven exploration. Evaluations on TED, TDT2, and MSR-VTT show strong multimodal performance and near state-of-the-art results without fine-tuning, with OpenAI embeddings achieving near-perfect MRR on TED tasks. The proposed UX combines exploration and exploitation in a 2D semantic map, enabling scalable, semantically informed video discovery with practical applicability across domains, while acknowledging limitations around indexing latency and privacy considerations.

Abstract

Streamlining content discovery within media archives requires integrating advanced data representations and effective visualization techniques for clear communication of video topics to users. The proposed system addresses the challenge of efficiently navigating large video collections by exploiting a fusion of visual, audio, and textual features to accurately index and categorize video content through a text-based method. Additionally, semantic embeddings are employed to provide contextually relevant information and recommendations to users, resulting in an intuitive and engaging exploratory experience over our topics ontology map using OpenAI GPT-4.

VCR: Video representation for Contextual Retrieval

TL;DR

VCR proposes a multimodal, text-based video representation framework for contextual retrieval by fusing ASR, OCR, and frame-caption signals into latent semantic embeddings indexed offline and queried online via GPT-4–generated descriptions. The approach includes a supervised multilabel classifier and a learning-free GPT-based encoder, both yielding a joint semantic space, and introduces the Topics-Map visualization to support ontology-driven exploration. Evaluations on TED, TDT2, and MSR-VTT show strong multimodal performance and near state-of-the-art results without fine-tuning, with OpenAI embeddings achieving near-perfect MRR on TED tasks. The proposed UX combines exploration and exploitation in a 2D semantic map, enabling scalable, semantically informed video discovery with practical applicability across domains, while acknowledging limitations around indexing latency and privacy considerations.

Abstract

Streamlining content discovery within media archives requires integrating advanced data representations and effective visualization techniques for clear communication of video topics to users. The proposed system addresses the challenge of efficiently navigating large video collections by exploiting a fusion of visual, audio, and textual features to accurately index and categorize video content through a text-based method. Additionally, semantic embeddings are employed to provide contextually relevant information and recommendations to users, resulting in an intuitive and engaging exploratory experience over our topics ontology map using OpenAI GPT-4.
Paper Structure (13 sections, 2 figures, 3 tables)

This paper contains 13 sections, 2 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Related work
  3. Multimodality
  4. Method
  5. A supervised approach
  6. A learning-free approach
  7. Topics ontology query generation
  8. The Topics-Map
  9. Implementation Details
  10. User Experience Design
  11. Evaluation
  12. User Study
  13. Discussion and Limitations

Figures (2)

  • Figure 1: Our Text-based multimodal video representation with LLMs.
  • Figure 2: Cosine-Similarities between TED descriptions and its corresponding ASR transcripts. The bold main diagonal illustrates the significance of our method in matching them.