Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

MMOU: A Massive Multi-Task Omni Understanding and Reasoning Benchmark for Long and Complex Real-World Videos

Arushi Goel, Sreyan Ghosh, Vatsal Agarwal, Nishit Anand, Kaousheik Jayakumar, Lasha Koroshinadze, Yao Xu, Katie Lyons, James Case, Karan Sapra, Kevin J. Shih, Siddharth Gururani, Abhinav Shrivastava, Ramani Duraiswami, Dinesh Manocha, Andrew Tao, Bryan Catanzaro, Mohammad Shoeybi, Wei Ping

Abstract

Multimodal Large Language Models (MLLMs) have shown strong performance in visual and audio understanding when evaluated in isolation. However, their ability to jointly reason over omni-modal (visual, audio, and textual) signals in long and complex videos remains largely unexplored. We introduce MMOU, a new benchmark designed to systematically evaluate multimodal understanding and reasoning under these challenging, real-world conditions. MMOU consists of 15,000 carefully curated questions paired with 9038 web-collected videos of varying length, spanning diverse domains and exhibiting rich, tightly coupled audio-visual content. The benchmark covers 13 fundamental skill categories, all of which require integrating evidence across modalities and time. All questions are manually annotated across multiple turns by professional annotators, ensuring high quality and reasoning fidelity. We evaluate 20+ state-of-the-art open-source and proprietary multimodal models on MMOU. The results expose substantial performance gaps: the best closed-source model achieves only 64.2% accuracy, while the strongest open-source model reaches just 46.8%. Our results highlight the challenges of long-form omni-modal understanding, revealing that current models frequently fail to apply even fundamental skills in long videos. Through detailed analysis, we further identify systematic failure modes and provide insights into where and why current models break.

MMOU: A Massive Multi-Task Omni Understanding and Reasoning Benchmark for Long and Complex Real-World Videos

Abstract

Multimodal Large Language Models (MLLMs) have shown strong performance in visual and audio understanding when evaluated in isolation. However, their ability to jointly reason over omni-modal (visual, audio, and textual) signals in long and complex videos remains largely unexplored. We introduce MMOU, a new benchmark designed to systematically evaluate multimodal understanding and reasoning under these challenging, real-world conditions. MMOU consists of 15,000 carefully curated questions paired with 9038 web-collected videos of varying length, spanning diverse domains and exhibiting rich, tightly coupled audio-visual content. The benchmark covers 13 fundamental skill categories, all of which require integrating evidence across modalities and time. All questions are manually annotated across multiple turns by professional annotators, ensuring high quality and reasoning fidelity. We evaluate 20+ state-of-the-art open-source and proprietary multimodal models on MMOU. The results expose substantial performance gaps: the best closed-source model achieves only 64.2% accuracy, while the strongest open-source model reaches just 46.8%. Our results highlight the challenges of long-form omni-modal understanding, revealing that current models frequently fail to apply even fundamental skills in long videos. Through detailed analysis, we further identify systematic failure modes and provide insights into where and why current models break.
Paper Structure (32 sections, 1 equation, 14 figures, 8 tables)

This paper contains 32 sections, 1 equation, 14 figures, 8 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. MMOU
  4. Overview
  5. Dataset Statistics
  6. Dataset Comparison
  7. Data Collection, Curation & Annotation
  8. Experimental Setup
  9. Baselines
  10. Evaluation
  11. Results and Discussion
  12. Results Analysis
  13. Conclusion, Limitations and Future Work
  14. Impact Statement
  15. Additional Dataset Statistics
  16. ...and 17 more sections

Figures (14)

  • Figure 1: Overview of MMOU, a benchmark for evaluating omni-modal understanding in long, complex real-world videos, showing that both open and closed multimodal models struggle even with basic understanding.
  • Figure 2: Illustrative examples from MMOU, demonstrating the different skill types evaluated in the benchmark.
  • Figure 3: Distribution of MMOU. (a) Video category distribution in MMOU, covering 10 major domains and 36 fine-grained subdomains. (b) Co-occurrence matrix of QA task types, illustrating how multiple reasoning skills are jointly required within individual questions. (c) Distribution of the relative temporal positions (average of start–end time-stamps) of answer evidence within videos, showing that answers are spread across the entire video timeline. (d) Distribution of QA instances across the 13 skill/task types in MMOU. (e) Video duration distribution, highlighting the prevalence of long and complex real-world videos.
  • Figure 4: Overview of the dataset-construction pipeline for MMOU.
  • Figure 5: Skill-wise performance comparison of various models on MMOU. Frontier models still struggle with basic skills like counting and finding temporal relationships between distinct events.
  • ...and 9 more figures