RTime-QA: A Benchmark for Atomic Temporal Event Understanding in Large Multi-modal Models

TL;DR

This work addresses a gap in evaluating temporal understanding in video-language systems by introducing RTime-QA, a benchmark of 822 temporally distinct QA items built from videos of atomic temporal events, each paired with temporally opposite descriptions in a multiple-choice format. To further enhance temporal reasoning, the authors provide RTime-IT, a 14,096-sample instruction-tuning dataset constructed via a parallel annotation pipeline, enabling targeted temporal training. Experiments show that RTime-QA is challenging for current large multi-modal systems, with Qwen2-VL achieving the best zero-shot performance but still far from human, especially under strict evaluation, while RTime-IT finetuning yields a substantial improvement in temporal understanding (Strict-ACC). The work emphasizes the importance of frame-level temporal cues, publicly vs privately sourced video data, and temporal-focused instruction tuning for advancing temporal semantics in video-language alignment, offering a practical pathway for dataset-driven progress. Overall, RTime-QA and RTime-IT together provide a rigorous framework for measuring and improving atomic temporal event understanding in video-language systems.

Abstract

Understanding accurate atomic temporal event is essential for video comprehension. However, current video-language benchmarks often fall short to evaluate Large Multi-modal Models' (LMMs) temporal event understanding capabilities, as they can be effectively addressed using image-language models. In this paper, we introduce RTime-QA, a novel benchmark specifically designed to assess the atomic temporal event understanding ability of LMMs. RTime-QA comprises 822 high-quality, carefully-curated video-text questions, each meticulously annotated by human experts. Each question features a video depicting an atomic temporal event, paired with both correct answers and temporal negative descriptions, specifically designed to evaluate temporal understanding. To advance LMMs' temporal event understanding ability, we further introduce RTime-IT, a 14k instruction-tuning dataset that employs a similar annotation process as RTime-QA. Extensive experimental analysis demonstrates that RTime-QA presents a significant challenge for LMMs: the state-of-the-art model Qwen2-VL achieves only 34.6 on strict-ACC metric, substantially lagging behind human performance. Furthermore, our experiments reveal that RTime-IT effectively enhance LMMs' capacity in temporal understanding. By fine-tuning on RTime-IT, our Qwen2-VL achieves 65.9 on RTime-QA.

Figures (4)

• Figure 1: Although the two videos share identical spatial appearances, they depict distinct atomic temporal events, which can only be differentiated through temporal understanding.
• Figure 2: The annotation pipeline of RTime-QA. We start by generating reference sentences with commercial LLMs. Next, we engage a team of human annotators to filter out data and write concise sentences. Finally, we structure the annotated video-sentence pairs into multiple-choice QA format.
• Figure 3: Examples of RTime-QA. The test samples in RTime-QA can be categorized into four types.
• Figure 4: Zero-shot response from different LMMs.