Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

QCaption: Video Captioning and Q&A through Fusion of Large Multimodal Models

Jiale Wang, Gee Wah Ng, Lee Onn Mak, Randall Cher, Ng Ding Hei Ryan, Davis Wang

TL;DR

QCaption addresses the challenge of unified video captioning and Q&A by fusing three specialized components in a late-fusion pipeline: a video-to-image key-frame extractor, an image-text LMM for per-frame analysis, and an LLM for text-based aggregation. The approach supports on-premises deployment and demonstrates substantial gains over baselines on YouCook2, MSR-VTT, and ActivityNet-QA, with improvements up to 44.2% for captioning and 48.9% for Q&A. Ablation studies reveal the critical role of the LLM in producing coherent, temporally-aware outputs and show how frame-sampling strategies affect performance across tasks. The work highlights the potential of model fusion to advance video analytics, offering a modular, swap-friendly framework that can incorporate newer LMMs/LLMs and additional modalities. Overall, QCaption establishes a practical baseline for fusion-based video captioning and Q&A that avoids costly retraining while delivering strong performance.

Abstract

This paper introduces QCaption, a novel video captioning and Q&A pipeline that enhances video analytics by fusing three models: key frame extraction, a Large Multimodal Model (LMM) for image-text analysis, and a Large Language Model (LLM) for text analysis. This approach enables integrated analysis of text, images, and video, achieving performance improvements over existing video captioning and Q&A models; all while remaining fully self-contained, adept for on-premises deployment. Experimental results using QCaption demonstrated up to 44.2% and 48.9% improvements in video captioning and Q&A tasks, respectively. Ablation studies were also performed to assess the role of LLM on the fusion on the results. Moreover, the paper proposes and evaluates additional video captioning approaches, benchmarking them against QCaption and existing methodologies. QCaption demonstrate the potential of adopting a model fusion approach in advancing video analytics.

QCaption: Video Captioning and Q&A through Fusion of Large Multimodal Models

TL;DR

QCaption addresses the challenge of unified video captioning and Q&A by fusing three specialized components in a late-fusion pipeline: a video-to-image key-frame extractor, an image-text LMM for per-frame analysis, and an LLM for text-based aggregation. The approach supports on-premises deployment and demonstrates substantial gains over baselines on YouCook2, MSR-VTT, and ActivityNet-QA, with improvements up to 44.2% for captioning and 48.9% for Q&A. Ablation studies reveal the critical role of the LLM in producing coherent, temporally-aware outputs and show how frame-sampling strategies affect performance across tasks. The work highlights the potential of model fusion to advance video analytics, offering a modular, swap-friendly framework that can incorporate newer LMMs/LLMs and additional modalities. Overall, QCaption establishes a practical baseline for fusion-based video captioning and Q&A that avoids costly retraining while delivering strong performance.

Abstract

This paper introduces QCaption, a novel video captioning and Q&A pipeline that enhances video analytics by fusing three models: key frame extraction, a Large Multimodal Model (LMM) for image-text analysis, and a Large Language Model (LLM) for text analysis. This approach enables integrated analysis of text, images, and video, achieving performance improvements over existing video captioning and Q&A models; all while remaining fully self-contained, adept for on-premises deployment. Experimental results using QCaption demonstrated up to 44.2% and 48.9% improvements in video captioning and Q&A tasks, respectively. Ablation studies were also performed to assess the role of LLM on the fusion on the results. Moreover, the paper proposes and evaluates additional video captioning approaches, benchmarking them against QCaption and existing methodologies. QCaption demonstrate the potential of adopting a model fusion approach in advancing video analytics.
Paper Structure (30 sections, 5 equations, 2 figures, 2 tables)

This paper contains 30 sections, 5 equations, 2 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Background
  3. Key contributions
  4. Related Works
  5. Pre-Large Large Language Models (LLM) era
  6. LLM
  7. Image Large Multimodal Models (LMMs)
  8. Video LMMs
  9. Fusion Techniques
  10. Methodology
  11. QCaption Fusion Architecture
  12. Key frame extraction
  13. Frame Captioning
  14. Caption Aggregation
  15. Advanced Sampling - Multi clips method
  16. ...and 15 more sections

Figures (2)

  • Figure 1: QCaption multi-modal fusion pipeline.
  • Figure 2: Comparison of Video-LLaVA and QCaption's video captioning capabilities