HumanVBench: Exploring Human-Centric Video Understanding Capabilities of MLLMs with Synthetic Benchmark Data

TL;DR

HumanVBench introduces a human-centric benchmark for video understanding in Multimodal Large Language Models, organizing 16 fine-grained tasks across inner emotion and outer manifestation. It couples an automated Human-Centric Video Annotation Pipeline with a Distractor-Included QA Synthesis Pipeline to synthesize scalable, high-quality data from wild videos, reducing manual labeling while maintaining quality through targeted human verification. A broad evaluation across 22 SOTA video MLLMs exposes persistent gaps in emotion perception and speech-visual alignment, with proprietary models often outperforming open-source counterparts but still failing to reach human-level performance in several cross-modal tasks. By open-sourcing the benchmark, data processing pipelines, and evaluation code, the work aims to drive rapid progress toward more human-like, robust video understanding in MLLMs.

Abstract

In the domain of Multimodal Large Language Models (MLLMs), achieving human-centric video understanding remains a formidable challenge. Existing benchmarks primarily emphasize object and action recognition, often neglecting the intricate nuances of human emotions, behaviors, and speech-visual alignment within video content. We present HumanVBench, an innovative benchmark meticulously crafted to bridge these gaps in the evaluation of video MLLMs. HumanVBench comprises 16 carefully designed tasks that explore two primary dimensions: inner emotion and outer manifestations, spanning static and dynamic, basic and complex, as well as single-modal and cross-modal aspects. With two advanced automated pipelines for video annotation and distractor-included QA generation, HumanVBench utilizes diverse state-of-the-art (SOTA) techniques to streamline benchmark data synthesis and quality assessment, minimizing human annotation dependency tailored to human-centric multimodal attributes. A comprehensive evaluation across 22 SOTA video MLLMs reveals notable limitations in current performance, especially in cross-modal and emotion perception, underscoring the necessity for further refinement toward achieving more human-like understanding. HumanVBench is open-sourced to facilitate future advancements and real-world applications in video MLLMs.

Figures (23)

• Figure 1: Overview of HumanVBench, which encompasses 16 fine-grained tasks for extensive human-centric evaluations (middle blue box). Each task is denoted by its acronym and the number of included QA instances. The right orange box illustrates some examples of these QAs. HumanVBench is constructed using the novel automated Video Annotation Pipeline (upper left, purple box), followed by the Distractor-Included QA Synthesis Pipeline (lower left, green box). These pipelines are reusable and backed by more than twenty data processing operators with advanced algorithm implementation and cutting-edge auxiliary models.
• Figure 2: The Human-Centric Video Annotation Pipeline involves obtaining videos featuring people and annotating both visual and auditory information as well as overall event atmospheres.
• Figure 3: The Distractor-Included QA Synthesis Pipeline facilitates four steps: selecting "question videos", generating preliminary answers, optimizing answers with generated distractors, and manually verifying multiple-choice questions.
• Figure 4: Two examples of 8-frame speaker videos sampled at equal intervals in the emotion recognition task, along with the responses from different MLLMs.
• Figure 5: Effectiveness in generating multiple-choice questions for six descriptive question types through the Distractor-Included QA Synthesis Pipeline.