Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

QMAVIS: Long Video-Audio Understanding using Fusion of Large Multimodal Models

Zixing Lin, Jiale Wang, Gee Wah Ng, Lee Onn Mak, Chan Zhi Yang Jeriel, Jun Yang Lee, Yaohao Li

TL;DR

QMAVIS tackles the gap in long-form video–audio understanding by integrating a video LMM, Whisper ASR, and an LLM via late fusion on chunked inputs. The method preserves temporal context through per-chunk captioning and transcription, interleaving results, and LLM-based aggregation to produce coherent long-form responses. It achieves substantial improvements on VideoMME (66.46 accuracy, 38.75% over VideoLlama2) and competitive gains on PerceptionTest and EgoSchema, with ablation studies confirming the contribution of each component. The approach enables on-premises deployment with open-source models, supporting privacy-sensitive applications in multimedia content analysis and sensemaking.

Abstract

Large Multimodal Models (LMMs) for video-audio understanding have traditionally been evaluated only on shorter videos of a few minutes long. In this paper, we introduce QMAVIS (Q Team-Multimodal Audio Video Intelligent Sensemaking), a novel long video-audio understanding pipeline built through a late fusion of LMMs, Large Language Models, and speech recognition models. QMAVIS addresses the gap in long-form video analytics, particularly for longer videos of a few minutes to beyond an hour long, opening up new potential applica- tions in sensemaking, video content analysis, embodied AI, etc. Quantitative experiments using QMAVIS demonstrated a 38.75% improvement over state-of-the-art video-audio LMMs like Vide- oLlaMA2 and InternVL2 on the VideoMME (with subtitles) dataset, which comprises long videos with audio information. Evaluations on other challenging video understanding datasets like PerceptionTest and EgoSchema saw up to 2% improvement, indicating competitive performance. Qualitative experiments also showed that QMAVIS is able to extract the nuances of different scenes in a long video audio content while understanding the overarching narrative. Ablation studies were also conducted to ascertain the impact of each component in the fusion pipeline.

QMAVIS: Long Video-Audio Understanding using Fusion of Large Multimodal Models

TL;DR

QMAVIS tackles the gap in long-form video–audio understanding by integrating a video LMM, Whisper ASR, and an LLM via late fusion on chunked inputs. The method preserves temporal context through per-chunk captioning and transcription, interleaving results, and LLM-based aggregation to produce coherent long-form responses. It achieves substantial improvements on VideoMME (66.46 accuracy, 38.75% over VideoLlama2) and competitive gains on PerceptionTest and EgoSchema, with ablation studies confirming the contribution of each component. The approach enables on-premises deployment with open-source models, supporting privacy-sensitive applications in multimedia content analysis and sensemaking.

Abstract

Large Multimodal Models (LMMs) for video-audio understanding have traditionally been evaluated only on shorter videos of a few minutes long. In this paper, we introduce QMAVIS (Q Team-Multimodal Audio Video Intelligent Sensemaking), a novel long video-audio understanding pipeline built through a late fusion of LMMs, Large Language Models, and speech recognition models. QMAVIS addresses the gap in long-form video analytics, particularly for longer videos of a few minutes to beyond an hour long, opening up new potential applica- tions in sensemaking, video content analysis, embodied AI, etc. Quantitative experiments using QMAVIS demonstrated a 38.75% improvement over state-of-the-art video-audio LMMs like Vide- oLlaMA2 and InternVL2 on the VideoMME (with subtitles) dataset, which comprises long videos with audio information. Evaluations on other challenging video understanding datasets like PerceptionTest and EgoSchema saw up to 2% improvement, indicating competitive performance. Qualitative experiments also showed that QMAVIS is able to extract the nuances of different scenes in a long video audio content while understanding the overarching narrative. Ablation studies were also conducted to ascertain the impact of each component in the fusion pipeline.
Paper Structure (23 sections, 5 equations, 2 figures, 1 table)

This paper contains 23 sections, 5 equations, 2 figures, 1 table.

Table of Contents

  1. Introduction
  2. Key Contributions
  3. Related Works and Fusion techniques
  4. Video LMM
  5. Video-audio LMM
  6. Methodology
  7. Fusion Architecture
  8. Video Chunking
  9. Video Captioning and Audio Transcription
  10. Caption Interleaving and Aggregation
  11. Ablation Studies
  12. Results
  13. Evaluation Metrics
  14. Benchmark Datasets
  15. Perception Test
  16. ...and 8 more sections

Figures (2)

  • Figure 1: QMAVIS Fusion Architecture.
  • Figure 2: Qualitative Results between QMAVIS, InternVideo2 PandaGPT and VideoLLaMA2