Integrated Semantic and Temporal Alignment for Interactive Video Retrieval

TL;DR

The paper tackles the challenge of interactive, real-world video retrieval under TRAKE by introducing a scalable modular framework that integrates TransNetV2 for scene segmentation, BEiT-3 for visual embeddings, and Gemini OCR for metadata, with Milvus as the vector index. It presents QUEST, a two-branch system that uses LLM-based query rewriting and external image grounding to overcome Out-of-Knowledge queries, and DANTE, a dynamic programming algorithm that efficiently aligns temporal event sequences to achieve coherent narrative retrieval. The authors demonstrate end-to-end capabilities across semantic, OCR, QUEST-enhanced, and temporally aligned DANTE queries, achieving strong performance in TRAKE and related tasks. The work offers a practical, extensible approach for robust, multimodal video search in real-world datasets, with potential impact on large-scale, interactive media retrieval systems.

Abstract

The growing volume of video data and the introduction of complex retrieval challenges, such as the Temporal Retrieval and Alignment of Key Events (TRAKE) task at the Ho Chi Minh City AI Challenge 2025, expose critical limitations in existing systems. Many methodologies lack scalable, holistic architectures and rely on "frozen" embedding models that fail on out-of-knowledge (OOK) or real-world queries. This paper introduces the comprehensive video retrieval framework developed by team AIO\_Owlgorithms to address these gaps. Our system features an architecture integrating TransNetV2 for scene segmentation, BEiT-3 for visual embeddings in Milvus, and Gemini OCR for metadata in Elasticsearch. We propose two components: (1) \textbf{QUEST} (Query Understanding and External Search for Out-of-Knowledge Tasks), a two-branch framework that leverages a Large Language Model (LLM) for query rewriting and an external image search pathway to resolve OOK queries; and (2) \textbf{DANTE} (Dynamic Alignment of Narrative Temporal Events), a dynamic programming algorithm that efficiently solves the temporally-incoherent TRAKE task. These contributions form a robust and intelligent system that significantly advances the state-of-the-art in handling complex, real-world video search queries.

Figures (9)

• Figure 1: Overview of the data preprocessing pipeline, including shot detection, frame sampling, metadata extraction, and feature embedding.
• Figure 2: Overview of the online retrieval pipeline, where text or image queries are processed by optional LLM enhancers, the BEiT-3 encoder, or specialized modules (DANTE, OCR) before ranked results are returned from Milvus or Elasticsearch.
• Figure 3: Overview of the QUEST framework.
• Figure 4: Illustration of DANTE workflow from input to output.
• Figure 5: Overview of the multi-modal search UI, comprising query input (A) with optional AI enhancer (E), search type and parameter controls (B), results display (C), and pre-search metadata filters (D); supports image upload or 'Find Similar' functionality within (C).