CURVE: A Benchmark for Cultural and Multilingual Long Video Reasoning

TL;DR

CURVE addresses the bias in existing video reasoning benchmarks by introducing a large-scale, multicultural, multilingual long-video QA dataset with fully human-curated questions, answers, and native-language reasoning traces across 18 locales. It stages a novel evaluation framework using evidence graphs and iterative error isolation to diagnose where Video-LLMs fail, revealing that most errors stem from culturally grounded visual perception rather than purely logical reasoning. The dataset comprises 2400 questions over 540 long videos spanning six cultural domains, offering audio in native languages and requiring complex temporal and multimodal reasoning. Experimental results show a substantial gap between human performance and current SOTA Video-LLMs, with audio contributing to gains and underrepresented languages bearing heavier cultural-perception burdens, underscoring the need for globally inclusive AI systems. CURVE also provides a rigorous, reproducible curation pipeline and a diagnostic methodology that can guide future improvements in fair, culturally aware AI models.

Abstract

Recent advancements in video models have shown tremendous progress, particularly in long video understanding. However, current benchmarks predominantly feature western-centric data and English as the dominant language, introducing significant biases in evaluation. To address this, we introduce CURVE (Cultural Understanding and Reasoning in Video Evaluation), a challenging benchmark for multicultural and multilingual video reasoning. CURVE comprises high-quality, entirely human-generated annotations from diverse, region-specific cultural videos across 18 global locales. Unlike prior work that relies on automatic translations, CURVE provides complex questions, answers, and multi-step reasoning steps, all crafted in native languages. Making progress on CURVE requires a deeply situated understanding of visual cultural context. Furthermore, we leverage CURVE's reasoning traces to construct evidence-based graphs and propose a novel iterative strategy using these graphs to identify fine-grained errors in reasoning. Our evaluations reveal that SoTA Video-LLMs struggle significantly, performing substantially below human-level accuracy, with errors primarily stemming from the visual perception of cultural elements. CURVE will be publicly available under https://github.com/google-deepmind/neptune?tab=readme-ov-file\#minerva-cultural

Figures (17)

• Figure 1: CURVE for benchmarking Cultural and Multilingual Video Reasoning. We show examples from four distinct global locales: en-GB (English, UK), es-MX (Spanish, Mexico), ta-IN (Tamil, India), ar-EG (Arabic, Egypt) within the CURVE benchmark. Each example consists of a video, a complex native-language question, the ground-truth answer, and two representative and incorrect SOTA VLM responses. We also show the associated reasoning skills and annotated human reasoning steps provided with each question.
• Figure 2: CURVE Benchmark Statistics.(a)CURVE contains a wide range of video durations, ranging from one minute to over an hour. (b) Human-authored reasoning traces are long and detailed, often spanning hundreds of words, while the corresponding answers are concise. (c) Questions are spread across six core cultural domains.
• Figure 3: The Human Annotation Pipeline. Four stages of our data curation process: Culture-specific video selection, 10% sample calibration, final data curation and audit (Section \ref{['sec:human_curation_pipeline']}), and human evaluation (Section \ref{['sec:human_eval']}). Each stage involves rigorous human curation and verification to ensure data quality and difficulty.
• Figure 4: Performance improvement when using both Audio and Video vs. Video only, on Gemini-2.5-pro. We see that adding audio consistently improves performance across most locales.
• Figure 5: Effect of increasing output token budget on Gemini-2.5-Pro's accuracy. Performance scales positively with compute, peaking at a 2k token budget followed by diminishing returns.