MINERVA: Evaluating Complex Video Reasoning

TL;DR

Minerva introduces a manually annotated, video reasoning benchmark with 1,515 questions, 5-choice answers, and detailed reasoning traces to reveal multi-step, cross-modal reasoning. By spanning diverse video domains and lengths, it challenges current multimodal models and enables analysis beyond final answers through a four-axis rubric (Perceptual Correctness, Temporal Localization, Logical Reasoning, Completeness) and MiRA LLM-based judgments. The study shows that temporal localization and visual perception are primary bottlenecks, and that rubric-informed prompting can improve both model performance and the interpretability of their reasoning traces. The dataset, baseline analyses, and MiRA framework lay groundwork for robust evaluation of genuine video reasoning and guide future improvements in multimodal AI systems.

Abstract

Multimodal LLMs are turning their focus to video benchmarks, however most video benchmarks only provide outcome supervision, with no intermediate or interpretable reasoning steps. This makes it challenging to assess if models are truly able to combine perceptual and temporal information to reason about videos, or simply get the correct answer by chance or by exploiting linguistic biases. To remedy this, we provide a new video reasoning dataset called MINERVA for modern multimodal models. Each question in the dataset comes with 5 answer choices, as well as detailed, hand-crafted reasoning traces. Our dataset is multimodal, diverse in terms of video domain and length, and consists of complex multi-step questions. Extensive benchmarking shows that our dataset provides a challenge for frontier open-source and proprietary models. We perform fine-grained error analysis to identify common failure modes across various models, and create a taxonomy of reasoning errors. We use this to explore both human and LLM-as-a-judge methods for scoring video reasoning traces, and find that failure modes are primarily related to temporal localization, followed by visual perception errors, as opposed to logical or completeness errors. The dataset, along with questions, answer candidates and reasoning traces will be publicly available under https://github.com/google-deepmind/neptune?tab=readme-ov-file\#minerva.

Figures (12)

• Figure 1: Examples from Minerva: We introduce Minerva, a complex video question-answering dataset. Unlike existing video datasets, the answer to each question is accompanied by a detailed reasoning trace, which outlines the steps required to come to the answer. Videos cover multiple domains such as (clockwise) -- sports, cooking, short films and science lectures. Reasoning traces are detailed, including timestamps (highlighted in green) and key actions (highlighted in pink). We show a single frame from each video.
• Figure 2: Dataset statistics. Video lengths (left), lengths of answers and reasoning (middle), and domains (right). Videos cover a wide range of lengths, with some longer than 100 minutes. Every question comes with a reasoning trace which is long and detailed, mean number of words is 92 (middle). Domains are hand-selected to include videos that lend themselves well to complex reasoning questions.
• Figure 3: Model and human accuracy (MCQ) broken down by (a) skill, (b) video domain, (c) video length. Note that questions can belong to more than one skill for (a). We provide a common legend for all 3 plots (in the right). Best viewed in color and with zoom. Axes in the radar charts are scaled to the highest accuracy per dimension. The full results for (a) are provided in Tab. \ref{['tab:mcq_scores_by_skill']} and for (b) in Tab. \ref{['tab:mcq_scores_by_category']} in the appendix.
• Figure 4: Analysis of model produced reasoning traces: We report MiRA scores on reasoning traces for two axes in the Minerva rubric. Models are presented in ascending order of scores.
• Figure 5: Direct MCQ prompt for Gemini.