LEMON: How Well Do MLLMs Perform Temporal Multimodal Understanding on Instructional Videos?

TL;DR

LEMON introduces a temporally grounded, multimodal lecture benchmark to evaluate STEM instructional videos. It collects 2,277 segments with synchronized video, audio, and subtitles, producing 4,181 QA pairs across six tasks and twelve subtasks that span perception, reasoning, and generation. Comprehensive experiments across 21 MLLMs reveal strong perceptual capabilities but persistent deficits in temporal/coherent reasoning, multimodal grounding, and cross-lingual generation, with proprietary models generally outperforming open-source ones. The benchmark emphasizes streaming, context-aware evaluation and aims to drive advances in robust, temporally grounded multimodal understanding for long-form instructional content.

Abstract

Recent multimodal large language models (MLLMs) have shown remarkable progress across vision, audio, and language tasks, yet their performance on long-form, knowledge-intensive, and temporally structured educational content remains largely unexplored. To bridge this gap, we introduce LEMON, a Lecture-based Evaluation benchmark for MultimOdal uNderstanding, focusing on STEM lecture videos that require long-horizon reasoning and cross-modal integration. LEMON comprises 2,277 video segments spanning 5 disciplines and 29 courses, with an average duration of 196.1 seconds, yielding 4,181 high-quality QA pairs, including 3,413 multiple-choice and 768 open-ended questions. Distinct from existing video benchmarks, LEMON features: (1) semantic richness and disciplinary density, (2) tightly coupled video-audio-text modalities, (3) explicit temporal and pedagogical structure, and (4) contextually linked multi-turn questioning. It further encompasses six major tasks and twelve subtasks, covering the full cognitive spectrum from perception to reasoning and then to generation. Comprehensive experiments reveal substantial performance gaps across tasks, highlighting that even state-of-the-art MLLMs like GPT-4o struggle with temporal reasoning and instructional prediction. We expect LEMON to serve as an extensible and challenging benchmark for advancing multimodal perception, reasoning, and generation in long-form instructional contents.

Figures (24)

• Figure 1: Characteristics of online lecture videos that motivate the design of LEMON. The teaching process naturally integrates multimodal cues (visual slides, speech, and text), covering from Perception (concept introduction) to Reasoning (example solution) and finally to Generation (knowledge summary).
• Figure 2: Statistics overview of LEMON. Left: Video categories included in LEMON. Top Right: Video duration distribution of LEMON. Bottom Right: Statistics on the number of all subtasks.
• Figure 3: Examples of LEMON. Each multiple-choice task forms a chain of interdependent subtasks, where each answer informs the next. The open-ended task assesses a model’s ability to produce coherent summaries and cross-lingual translations.
• Figure 4: Analysis of model performance across languages and tasks. (a) Average performance on different languages measured by BLEUpapineni-etal-2002-bleu, ROUGE-Llin-2004-rouge, and BERTScorezhang2019bertscore. ZH: Chinese, JA: Japanese, KO: Korean, FR: French, DE: German, RU: Russian, ES: Spanish, AR: Arabic. (b) Task performance comparison shows strengths in Perception and weaknesses in Reasoning and Generation.
• Figure 5: Impact of frame sampling and video duration on Streaming Perception performance. Accuracy decreases with longer videos under sparse sampling, while dense sampling improves performance for longer clips but introduces redundancy in short ones.