RTV-Bench: Benchmarking MLLM Continuous Perception, Understanding and Reasoning through Real-Time Video

TL;DR

RTV-Bench addresses the gap in evaluating multimodal large language models on continuous, real-time video analysis. It introduces Multi-Timestamp QA, a hierarchical question structure, and a multidimensional evaluation framework to assess perception, understanding, and reasoning as events unfold. Experimental results show online real-time models outperform offline counterparts, while increasing model size or frame sampling yields limited gains, highlighting the need for architectures optimized for streaming video and long sequences. The benchmark toolkit is openly available, offering a platform to push forward robust online video understanding in practical applications like autonomous driving and live analytics.

Abstract

Multimodal Large Language Models (MLLMs) increasingly excel at perception, understanding, and reasoning. However, current benchmarks inadequately evaluate their ability to perform these tasks continuously in dynamic, real-world environments. To bridge this gap, we introduce RTV-Bench, a fine-grained benchmark for MLLM real-time video analysis. RTV-Bench uses three key principles: (1) Multi-Timestamp Question Answering (MTQA), where answers evolve with scene changes; (2) Hierarchical Question Structure, combining basic and advanced queries; and (3) Multi-dimensional Evaluation, assessing the ability of continuous perception, understanding, and reasoning. RTV-Bench contains 552 diverse videos (167.2 hours) and 4,631 high-quality QA pairs. We evaluated leading MLLMs, including proprietary (GPT-4o, Gemini 2.0), open-source offline (Qwen2.5-VL, VideoLLaMA3), and open-source real-time (VITA-1.5, InternLM-XComposer2.5-OmniLive) models. Experiment results show open-source real-time models largely outperform offline ones but still trail top proprietary models. Our analysis also reveals that larger model size or higher frame sampling rates do not significantly boost RTV-Bench performance, sometimes causing slight decreases. This underscores the need for better model architectures optimized for video stream processing and long sequences to advance real-time video analysis with MLLMs. Our benchmark toolkit is available at: https://github.com/LJungang/RTV-Bench.

Figures (10)

• Figure 1: Specific examples of different tasks in RTV-Bench. and represent the fundamental questions in the question group, and the answers are underlined. is a dynamically answered question, and the correct answer is determined by the time of the query. The correct answer to the query will change at different times, and we have marked the correct answers that should be responded to at different query times.
• Figure 2: Video Categories and Distribution of Question Difficulty and Query Characteristics. (Left) RTV-Bench overs 3 key domains and 16 sub-class video types. (Center) Distribution of question difficulty levels across eight representative task types, measured by percentage-based performance ranges. (Right) Distribution of question queries by video length, categorized into Shallow, Moderate, and Deep levels. The bar heights indicate counts, while the line chart overlays query proportions for each duration bucket.
• Figure 3: Performance visualization and analysis on RTV-Bench: (a) Visualization of overall Accuracy results; (b) Visualization of overall Score results; (c) Performance impact of varying input frame counts; (d) Performance comparison across different Qwen2.5-VL model scales.
• Figure 4: Comparison of response from different models for the same question on the same video. Green indicates correct answers or timestamps; red indicates incorrect answers or timestamps. This case demonstrates that even current high-performance models struggle to provide high-quality responses (where both the answer and the corresponding timestamp are accurate).
• Figure B.1: Our dataset construction pipeline. We develop a dataset generation pipeline consisting of three stages to create RTV-Bench: Video Collection, Templates Design and Human Annotation.