SONIC-O1: A Real-World Benchmark for Evaluating Multimodal Large Language Models on Audio-Video Understanding

TL;DR

SONIC-O1 presents a real-world, open, human-verified benchmark for evaluating multimodal LLMs on audio–video understanding across 13 topics and five domains. It defines three tasks—video summarization, MCQ with reasoning, and temporal localization with rationales—evaluated on nearly 5,000 instances derived from ~60 hours of content, with demographic metadata enabling group-wise fairness analysis. The study reveals that closed-source models generally outperform open-source counterparts, temporal localization remains the most challenging task for omnimodal models, and substantial demographic disparities persist, especially in race and age. The authors provide a comprehensive evaluation suite (data, scripts, leaderboard) to foster reproducibility and community-driven progress toward more temporally grounded and socially robust MLLMs.

Abstract

Multimodal Large Language Models (MLLMs) are a major focus of recent AI research. However, most prior work focuses on static image understanding, while their ability to process sequential audio-video data remains underexplored. This gap highlights the need for a high-quality benchmark to systematically evaluate MLLM performance in a real-world setting. We introduce SONIC-O1, a comprehensive, fully human-verified benchmark spanning 13 real-world conversational domains with 4,958 annotations and demographic metadata. SONIC-O1 evaluates MLLMs on key tasks, including open-ended summarization, multiple-choice question (MCQ) answering, and temporal localization with supporting rationales (reasoning). Experiments on closed- and open-source models reveal limitations. While the performance gap in MCQ accuracy between two model families is relatively small, we observe a substantial 22.6% performance difference in temporal localization between the best performing closed-source and open-source models. Performance further degrades across demographic groups, indicating persistent disparities in model behavior. Overall, SONIC-O1 provides an open evaluation suite for temporally grounded and socially robust multimodal understanding. We release SONIC-O1 for reproducibility and research: Project page: https://vectorinstitute.github.io/sonic-o1/ Dataset: https://huggingface.co/datasets/vector-institute/sonic-o1 Github: https://github.com/vectorinstitute/sonic-o1 Leaderboard: https://huggingface.co/spaces/vector-institute/sonic-o1-leaderboard

Figures (14)

• Figure 1: Performance comparison across 13 conversational domains. We compare closed-source and open-source MLLMs across 13 conversational domains using LLM-judge scores (0–10) for video summarization task. Gemini 3.0 Pro consistently outperforms open-source models, and high-stakes domains (e.g., Emergency Response, Mental Health) remain more challenging.
• Figure 2: Overview of SONIC-O1 tasks and evaluation format: (Top) video summarization, (Middle) evidence-grounded MCQ, (Bottom) temporal localization (event timing). Each example shows the input clip (frames) and the expected output format; demographic attributes shown beneath each clip represent associated metadata, enabling group-wise evaluation across 13 domains.
• Figure 3: Video categories. Our benchmark covers 5 key domains and 13 video topics.
• Figure 4: Video duration and question type distributions. SONIC-O1 spans a full spectrum of video lengths and evaluates core abilities of MLLMs.
• Figure 5: Frame-count sensitivity across metrics. Performance of three representative MLLMs (Qwen3-Omni, UniMoE-2, VideoLLaMA2) on representative topics across varying frame counts (16, 32, 64, 128) for three tasks: (a) Summarization quality measured by LLM judge scores, (b) Multiple-choice question accuracy, and (c) Temporal localization R@0.5. Models exhibit different saturation patterns across tasks.