LongInsightBench: A Comprehensive Benchmark for Evaluating Omni-Modal Models on Human-Centric Long-Video Understanding

TL;DR

LongInsightBench introduces the first benchmark tailored for long-video omni-modal understanding, combining visual, audio, and textual modalities across ~1000 high-density videos from FineVideo to probe temporal localization, long-range causal reasoning, and cross-modal fusion. The dataset employs a rigorous three-stage QA pipeline and six task types that span intra-event and inter-event reasoning, enabling systematic evaluation of Omni-modal Models (OLMs) alongside VLMs, ALMs, and LLMs. Experimental results reveal a persistent fusion deficit in current OLMs, with textual proxies often outperforming raw multimodal fusion, and show that increasing frame sampling can improve performance in a model-dependent manner. The work also discusses limitations such as API costs for high-fidelity QA generation and outlines future directions toward more cost-efficient, open-source pipelines for scalable long-video multimodal evaluation.

Abstract

We introduce \textbf{LongInsightBench}, the first benchmark designed to assess models' ability to understand long videos, with a focus on human language, viewpoints, actions, and other contextual elements, while integrating \textbf{visual, audio, and text} modalities. Our benchmark excels in three key areas: \textbf{a) Long-Duration, Information-Dense Videos:} We carefully select approximately 1,000 videos from open-source datasets FineVideo based on duration limit and the information density of both visual and audio modalities, focusing on content like lectures, interviews, and vlogs, which contain rich language elements. \textbf{b) Diverse and Challenging Task Scenarios:} We have designed six challenging task scenarios, including both Intra-Event and Inter-Event Tasks. \textbf{c) Rigorous and Comprehensive Quality Assurance Pipelines:} We have developed a three-step, semi-automated data quality assurance pipeline to ensure the difficulty and validity of the synthesized questions and answer options. Based on LongInsightBench, we designed a series of experiments. Experimental results shows that Omni-modal models(OLMs) still face challenge in tasks requiring precise temporal localization (T-Loc) and long-range causal inference (CE-Caus). Extended experiments reveal the information loss and processing bias in multi-modal fusion of OLMs. Our dataset and code is available at https://anonymous.4open.science/r/LongInsightBench-910F/.

Figures (16)

• Figure 1: Task Samples in LongInsightBench. The upper one comes from IE-Rea(Intra-Event Reasoning) subcategory and the lower one comes from T-Recon(Timeline Reconstruction) subcategory.
• Figure 2: Overview of the LongInsightBench construction workflow. The pipeline begins with video selection from FineVideo, applying filters on duration, scene shifts, and content richness. Next, automated annotation integrates visual and audio descriptions via MLLMs. These annotations support task scenario design and question generation, spanning intra-event and inter-event reasoning tasks. Finally, a quality assurance process combines automatic filtering, scoring, and manual validation to ensure a high-quality QA set.
• Figure 3: Fine-grained performance across task categories. Different OLMs’ accuracies are shown over six question types, highlighting each model’s strengths and weaknesses across categories.
• Figure 4: The distribution of videos in LongInsightBench across all subcategories.
• Figure 5: Prompt used for providing visual captions for each video.