Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

RAVEN: An Agentic Framework for Multimodal Entity Discovery from Large-Scale Video Collections

Kevin Dela Rosa

TL;DR

RAVEN addresses the challenge of organizing and retrieving information from large-scale, multimodal video collections by introducing a model-agnostic, agentic framework that combines vision-language processing with large-language reasoning. The approach splits into two flows: Category Understanding & Schema Generation, which derives canonical categories and domain-specific entity schemas, and Rich Domain Specific Entity Extraction, which performs schema-guided, in-context extraction of entities and attributes across videos. Key contributions include the modular architecture for category canonicalization, a dynamic schema generation mechanism, and a schema-guided extraction pipeline that demonstrates strong performance on large-scale video data with off-the-shelf LLMs/VLMs. The results show that RAVEN can generate structured, domain-adaptive representations and improve recall over unimodal baselines, enabling scalable, precise retrieval across vast video datasets. This framework has practical impact for personalized search, content discovery, and scalable multimodal information retrieval in diverse video corpora.

Abstract

We present RAVEN an adaptive AI agent framework designed for multimodal entity discovery and retrieval in large-scale video collections. Synthesizing information across visual, audio, and textual modalities, RAVEN autonomously processes video data to produce structured, actionable representations for downstream tasks. Key contributions include (1) a category understanding step to infer video themes and general-purpose entities, (2) a schema generation mechanism that dynamically defines domain-specific entities and attributes, and (3) a rich entity extraction process that leverages semantic retrieval and schema-guided prompting. RAVEN is designed to be model-agnostic, allowing the integration of different vision-language models (VLMs) and large language models (LLMs) based on application-specific requirements. This flexibility supports diverse applications in personalized search, content discovery, and scalable information retrieval, enabling practical applications across vast datasets.

RAVEN: An Agentic Framework for Multimodal Entity Discovery from Large-Scale Video Collections

TL;DR

RAVEN addresses the challenge of organizing and retrieving information from large-scale, multimodal video collections by introducing a model-agnostic, agentic framework that combines vision-language processing with large-language reasoning. The approach splits into two flows: Category Understanding & Schema Generation, which derives canonical categories and domain-specific entity schemas, and Rich Domain Specific Entity Extraction, which performs schema-guided, in-context extraction of entities and attributes across videos. Key contributions include the modular architecture for category canonicalization, a dynamic schema generation mechanism, and a schema-guided extraction pipeline that demonstrates strong performance on large-scale video data with off-the-shelf LLMs/VLMs. The results show that RAVEN can generate structured, domain-adaptive representations and improve recall over unimodal baselines, enabling scalable, precise retrieval across vast video datasets. This framework has practical impact for personalized search, content discovery, and scalable multimodal information retrieval in diverse video corpora.

Abstract

We present RAVEN an adaptive AI agent framework designed for multimodal entity discovery and retrieval in large-scale video collections. Synthesizing information across visual, audio, and textual modalities, RAVEN autonomously processes video data to produce structured, actionable representations for downstream tasks. Key contributions include (1) a category understanding step to infer video themes and general-purpose entities, (2) a schema generation mechanism that dynamically defines domain-specific entities and attributes, and (3) a rich entity extraction process that leverages semantic retrieval and schema-guided prompting. RAVEN is designed to be model-agnostic, allowing the integration of different vision-language models (VLMs) and large language models (LLMs) based on application-specific requirements. This flexibility supports diverse applications in personalized search, content discovery, and scalable information retrieval, enabling practical applications across vast datasets.

Paper Structure

This paper contains 8 sections, 5 figures, 2 tables.

Table of Contents

  1. Introduction
  2. RAVEN Framework Overview
  3. Category Understanding & Schema Generation
  4. Rich Domain Specific Entity Extraction
  5. Experiments & Evaluation
  6. Category Understanding & Schema Generation Analysis
  7. Domain Specific Entity Richness Analysis
  8. Conclusion

Figures (5)

  • Figure 1: RAVEN Framework Overview. The two main LLM / VLM driven flows: 1) Category Understanding & Schema Generation, 2) Rich Domain Specific Entity Extraction
  • Figure 2: Inferred canonical video category distribution
  • Figure 3: Distribution of extracted structured attributes for each generic entity type
  • Figure 4: Distribution of domain-specific entity types & attributes for How-To and History videos
  • Figure 5: Entity Recall by method. Each bar represents the proportion of entities extracted by each method for the Person, Location, and Object entity types respectively