SeViCES: Unifying Semantic-Visual Evidence Consensus for Long Video Understanding

TL;DR

SeViCES tackles the challenge of long video understanding by introducing a training-free, model-agnostic framework that unifies semantic and visual evidence through a two-stage consensus approach. The Semantic-Visual Consensus Frame Selection (SVCFS) comprises a Temporal-Aware Semantic Frame Selection and a Cluster-guided Mutual Information Frame Selection to produce a query-relevant, evidence-complete frame set; the Answer Consensus Refinement (ACR) then reconciles semantic- and visual-based predictions by fuse-evidence and constrained decoding. Across four long-video benchmarks and multiple Video-LLMs, SeViCES consistently improves accuracy and robustness, with average gains around 4% and peaks up to 8.3%. The work demonstrates the importance of consensus-driven frame selection and answer-level refinement for scalable, reliable long-video reasoning in real-world applications.

Abstract

Long video understanding remains challenging due to its complex, diverse, and temporally scattered content. Although video large language models (Video-LLMs) can process videos lasting tens of minutes, applying them to truly long sequences is computationally prohibitive and often leads to unfocused or inconsistent reasoning. A promising solution is to select only the most informative frames, yet existing approaches typically ignore temporal dependencies or rely on unimodal evidence, limiting their ability to provide complete and query-relevant context. We propose a Semantic-Visual Consensus Evidence Selection (SeViCES) framework for effective and reliable long video understanding. SeViCES is training-free and model-agnostic, and introduces two key components. The Semantic-Visual Consensus Frame Selection (SVCFS) module selects frames through (1) a temporal-aware semantic branch that leverages LLM reasoning over captions, and (2) a cluster-guided visual branch that aligns embeddings with semantic scores via mutual information. The Answer Consensus Refinement (ACR) module further resolves inconsistencies between semantic- and visual-based predictions by fusing evidence and constraining the answer space. Extensive experiments on long video understanding benchmarks show that SeViCES consistently outperforms state-of-the-art methods in both accuracy and robustness, demonstrating the importance of consensus-driven evidence selection for Video-LLMs.

Figures (5)

• Figure 1: The overall framework of our SeViCES. Video frames are processed by BLIP-2 to obtain captions and embeddings. TAS-FS selects query-relevant frames from captions, while CgMI-FS selects frames from embeddings. The two sets are then refined by the ACR module through evidence fusion and constrained decoding to produce the final answer.
• Figure 2: Performance of LLaVA-Video and Qwen2.5-VL on LVBench tasks with and without SeViCES. Task types include Key Information Retrieval (KIR), Event Understanding (EU), Entity Recognition (ER), Reasoning (Rea), and Temporal Grounding (TG).
• Figure 2: Performance changes of different SeViCES components.
• Figure 3: Performance changes with different numbers of selected frames.
• Figure 4: Two representative VideoQA task examples using Qwen2.5-VL and VideoMME dataset.