Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Enhanced Multimodal Video Retrieval System: Integrating Query Expansion and Cross-modal Temporal Event Retrieval

Van-Thinh Vo, Minh-Khoi Nguyen, Minh-Huy Tran, Anh-Quan Nguyen-Tran, Duy-Tan Nguyen, Khanh-Loi Nguyen, Anh-Minh Phan

TL;DR

The paper tackles robust, temporally-aware video retrieval across multiple modalities by introducing an adaptive KDE-GMM thresholding method for scene boundary detection, a cross-modal temporal event retrieval framework, and LLM-based query augmentation via Gemini. It combines embedding-based and metadata-based search with a Reciprocal Rank Fusion re-ranking strategy and a cross-modal temporal reasoning module that selects a pivot scene and optimizes over a 10-frame window. Key contributions include adaptive keyframe extraction, rich multimodal metadata, and a unified UI to support complex multi-scene queries, demonstrated in the Ho Chi Minh AI Challenge 2025. The approach promises improved precision and efficiency for large-scale, real-world video search tasks, with broad applicability to surveillance, content discovery, and multimedia databases.

Abstract

Multimedia information retrieval from videos remains a challenging problem. While recent systems have advanced multimodal search through semantic, object, and OCR queries - and can retrieve temporally consecutive scenes - they often rely on a single query modality for an entire sequence, limiting robustness in complex temporal contexts. To overcome this, we propose a cross-modal temporal event retrieval framework that enables different query modalities to describe distinct scenes within a sequence. To determine decision thresholds for scene transition and slide change adaptively, we build Kernel Density Gaussian Mixture Thresholding (KDE-GMM) algorithm, ensuring optimal keyframe selection. These extracted keyframes act as compact, high-quality visual exemplars that retain each segment's semantic essence, improving retrieval precision and efficiency. Additionally, the system incorporates a large language model (LLM) to refine and expand user queries, enhancing overall retrieval performance. The proposed system's effectiveness and robustness were demonstrated through its strong results in the Ho Chi Minh AI Challenge 2025.

Enhanced Multimodal Video Retrieval System: Integrating Query Expansion and Cross-modal Temporal Event Retrieval

TL;DR

The paper tackles robust, temporally-aware video retrieval across multiple modalities by introducing an adaptive KDE-GMM thresholding method for scene boundary detection, a cross-modal temporal event retrieval framework, and LLM-based query augmentation via Gemini. It combines embedding-based and metadata-based search with a Reciprocal Rank Fusion re-ranking strategy and a cross-modal temporal reasoning module that selects a pivot scene and optimizes over a 10-frame window. Key contributions include adaptive keyframe extraction, rich multimodal metadata, and a unified UI to support complex multi-scene queries, demonstrated in the Ho Chi Minh AI Challenge 2025. The approach promises improved precision and efficiency for large-scale, real-world video search tasks, with broad applicability to surveillance, content discovery, and multimedia databases.

Abstract

Multimedia information retrieval from videos remains a challenging problem. While recent systems have advanced multimodal search through semantic, object, and OCR queries - and can retrieve temporally consecutive scenes - they often rely on a single query modality for an entire sequence, limiting robustness in complex temporal contexts. To overcome this, we propose a cross-modal temporal event retrieval framework that enables different query modalities to describe distinct scenes within a sequence. To determine decision thresholds for scene transition and slide change adaptively, we build Kernel Density Gaussian Mixture Thresholding (KDE-GMM) algorithm, ensuring optimal keyframe selection. These extracted keyframes act as compact, high-quality visual exemplars that retain each segment's semantic essence, improving retrieval precision and efficiency. Additionally, the system incorporates a large language model (LLM) to refine and expand user queries, enhancing overall retrieval performance. The proposed system's effectiveness and robustness were demonstrated through its strong results in the Ho Chi Minh AI Challenge 2025.

Paper Structure

This paper contains 27 sections, 1 equation, 6 figures, 2 algorithms.

Table of Contents

  1. Introduction
  2. Related Work
  3. Importance and Applications of Video Retrieval
  4. Existing Systems and Innovations
  5. Challenges in Temporal Queries and Proposed Strategy
  6. Method
  7. Data Preprocessing
  8. Frame Extraction:
  9. Visual Feature Extraction:
  10. Metadata Representation:
  11. System Overall
  12. Embedding-based Search:
  13. Metadata-based Search:
  14. Re-ranking:
  15. Multimodal Search:
  16. ...and 12 more sections

Figures (6)

  • Figure 1: Overview of the data preprocessing pipeline. Raw videos are first processed to extract representative keyframes stored in the Keyframe Database. Each keyframe is then analyzed through BEiT3/CLIP feature extraction, OCR, object detection, and color detection modules. The outputs are consolidated into structured metadata for downstream multimodal retrieval.
  • Figure 2: After detecting objects in the frame, we encode them into a string of text-based bounding boxs as above. In addition, the other metadata collected are the object tags and counts, e.g. "person1 car1".
  • Figure 3: The Overview of Retrieval System. We propose two main search engines: Embedding-based Search and Metadata-based Search. To enhance the system performance, we also integrate them to address multimodal query and temporal events retrieval.
  • Figure 4: The user interface
  • Figure 5: Retrieval results without and with query augmentation.
  • ...and 1 more figures