LiViBench: An Omnimodal Benchmark for Interactive Livestream Video Understanding

TL;DR

LiViBench addresses the gap in evaluating interactive livestream video understanding by introducing the first omnimodal livestream benchmark, encompassing audio, speech, and real-time comments across 24 tasks in 9 domains. The authors propose a standardized semi-automatic annotation workflow using a multi-agent QA generation pipeline and a seed-question-driven strategy, plus a Video-to-Comment Retrieval module and a tailored two-stage instruction-tuning regime to build LiVi-LLM-7B. Empirical results show LiVi-LLM-7B and seed-based models outperform many open-source competitors with similar scales and approach proprietary performance on LiViBench, while also generalizing well to other general video benchmarks. The work provides practical baselines and insights for interactive livestream understanding, highlighting the value of multimodal context, domain-specific instruction tuning, and comment-aware retrieval in real-time video comprehension.

Abstract

The development of multimodal large language models (MLLMs) has advanced general video understanding. However, existing video evaluation benchmarks primarily focus on non-interactive videos, such as movies and recordings. To fill this gap, this paper proposes the first omnimodal benchmark for interactive livestream videos, LiViBench. It features a diverse set of 24 tasks, highlighting the perceptual, reasoning, and livestream-specific challenges. To efficiently construct the dataset, we design a standardized semi-automatic annotation workflow that incorporates the human-in-the-loop at multiple stages. The workflow leverages multiple MLLMs to form a multi-agent system for comprehensive video description and uses a seed-question-driven method to construct high-quality annotations. All interactive videos in the benchmark include audio, speech, and real-time comments modalities. To enhance models' understanding of interactive videos, we design tailored two-stage instruction-tuning and propose a Video-to-Comment Retrieval (VCR) module to improve the model's ability to utilize real-time comments. Based on these advancements, we develop LiVi-LLM-7B, an MLLM with enhanced knowledge of interactive livestreams. Experiments show that our model outperforms larger open-source models with up to 72B parameters, narrows the gap with leading proprietary models on LiViBench, and achieves enhanced performance on general video benchmarks, including VideoMME, LongVideoBench, MLVU, and VideoEval-Pro.

Figures (10)

• Figure 1: Dataset generation pipeline.
• Figure 2: The statistical analysis of our LiViBench.
• Figure 3: The ASR and comment distribution.
• Figure 4: The word clouds of our LiViBench.
• Figure 5: LiVi-LLM architecture. (a) Training process: In the first stage, the model is aligned to the interactive video domain using synthetic data; in the second stage, it undergoes fine-grained tuning with manual data. (b) Inference process: The model integrates a video-to-comment retrieval module to fully use omni-modality to enhance the comprehensive understanding.