VideoAutoArena: An Automated Arena for Evaluating Large Multimodal Models in Video Analysis through User Simulation

TL;DR

The paper tackles the gap between real-world user needs and traditional video benchmarks by introducing VideoAutoArena, an automated, arena-style framework that uses user simulation, peer battles, automated judging, and fault-driven evolution to evaluate large multimodal models in video analysis at scale. It demonstrates strong alignment with human judgments and realistic user-question styles, while revealing substantial gaps between open-source LMMs and GPT-4o, especially on longer videos. Complementing this, VideoAutoBench provides a faster, human-grounded evaluation by comparing model outputs to human-selected answers using GPT-4o as judge. Collectively, these contributions offer a cost-effective, scalable, and user-centric approach to advancing video understanding in LMMs and point to future work in multiturn and multilingual contexts as well as fairness in automatic judging.

Abstract

Large multimodal models (LMMs) with advanced video analysis capabilities have recently garnered significant attention. However, most evaluations rely on traditional methods like multiple-choice questions in benchmarks such as VideoMME and LongVideoBench, which are prone to lack the depth needed to capture the complex demands of real-world users. To address this limitation-and due to the prohibitive cost and slow pace of human annotation for video tasks-we introduce VideoAutoArena, an arena-style benchmark inspired by LMSYS Chatbot Arena's framework, designed to automatically assess LMMs' video analysis abilities. VideoAutoArena utilizes user simulation to generate open-ended, adaptive questions that rigorously assess model performance in video understanding. The benchmark features an automated, scalable evaluation framework, incorporating a modified ELO Rating System for fair and continuous comparisons across multiple LMMs. To validate our automated judging system, we construct a 'gold standard' using a carefully curated subset of human annotations, demonstrating that our arena strongly aligns with human judgment while maintaining scalability. Additionally, we introduce a fault-driven evolution strategy, progressively increasing question complexity to push models toward handling more challenging video analysis scenarios. Experimental results demonstrate that VideoAutoArena effectively differentiates among state-of-the-art LMMs, providing insights into model strengths and areas for improvement. To further streamline our evaluation, we introduce VideoAutoBench as an auxiliary benchmark, where human annotators label winners in a subset of VideoAutoArena battles. We use GPT-4o as a judge to compare responses against these human-validated answers. Together, VideoAutoArena and VideoAutoBench offer a cost-effective, and scalable framework for evaluating LMMs in user-centric video analysis.

Figures (18)

• Figure 1: Video statistics by category and duration.
• Figure 2: Examples of synthesized personas with three levels of relevance and corresponding synthesized questions. We also compare the style of our questions with those in popular long-video benchmarks, including LongVideoBench and VideoMME.
• Figure 3: Our user simulation offers diverse personas and more effectively mirrors real-world users' question styles.
• Figure 4: Our fault-driven evolution strategy generates increasingly challenging questions for video analysis.
• Figure 5: Evaluate the accuracy of various judging methods using human annotations as the gold standard. In the Vote (Top N) method, the top N models are used to cast votes.