Realizing Video Summarization from the Path of Language-based Semantic Understanding

TL;DR

The paper addresses scalable, semantically rich video summarization amid proliferating video content. It introduces an inference-time Mixture of Experts framework that coordinates multiple VideoLLMs to produce comprehensive textual summaries without fine-tuning. A denoise-and-cooperate pipeline with outlier filtering and flexible fusion strategies, plus a CLIP-based keyframe retrieval module, enables high-quality summaries and robust keyframe extraction, including audio-visual grounding. Extended applications in visual manual generation and privacy-preserving content generation demonstrate practical utility, with experiments showing strong cross-dataset performance and adaptability to new VideoLLMs.

Abstract

The recent development of Video-based Large Language Models (VideoLLMs), has significantly advanced video summarization by aligning video features and, in some cases, audio features with Large Language Models (LLMs). Each of these VideoLLMs possesses unique strengths and weaknesses. Many recent methods have required extensive fine-tuning to overcome the limitations of these models, which can be resource-intensive. In this work, we observe that the strengths of one VideoLLM can complement the weaknesses of another. Leveraging this insight, we propose a novel video summarization framework inspired by the Mixture of Experts (MoE) paradigm, which operates as an inference-time algorithm without requiring any form of fine-tuning. Our approach integrates multiple VideoLLMs to generate comprehensive and coherent textual summaries. It effectively combines visual and audio content, provides detailed background descriptions, and excels at identifying keyframes, which enables more semantically meaningful retrieval compared to traditional computer vision approaches that rely solely on visual information, all without the need for additional fine-tuning. Moreover, the resulting summaries enhance performance in downstream tasks such as summary video generation, either through keyframe selection or in combination with text-to-image models. Our language-driven approach offers a semantically rich alternative to conventional methods and provides flexibility to incorporate newer VideoLLMs, enhancing adaptability and performance in video summarization tasks.

Figures (8)

• Figure 2: An overview of our framework. Our approach consists of three main modules: (1) Video Summarization, which constructs coherent textual summaries by leveraging multiple existing VideoLLMs and our proposed inference-time algorithm; (2) Keyframe Retrieval, which identifies key moments based on our textual summary using a simple keyframe selection algorithm; and (3) Extended Applications, which utilize our informative textual summaries and keyframes to address real-world tasks beyond traditional video summarization.
• Figure 3: Visualization of textual video summaries generated by individual VideoLLMs and our proposed collaboration approach. Keyframes are displayed at the top as input video. Additionally, we provide the ground truth summary for reference.
• Figure 4: Visualization of prediction results comparison on QVHighlights lei2021detecting. The ground truth keyframes are shown at the top as the input video, and the prediction unit is in seconds.
• Figure 5: Prompt template of "Find common ground" strategy in the cooperation step.
• Figure 6: Prompt template of "Merge" strategy in the cooperation step.