ActionArt: Advancing Multimodal Large Models for Fine-Grained Human-Centric Video Understanding

TL;DR

ActionArt introduces a fine-grained, human-centric video understanding benchmark built on MoVid, with seven spatial-temporal subtasks and 2,678 QA pairs. It tackles the data bottleneck by proposing three proxy tasks—pose description, spatial differences mining, and synthesized fine-grained QA—whose data can be automatically generated from existing MLLMs, reducing reliance on manual labeling. Experiments show current MLLMs struggle with fine-grained interpretation, but proxy-task learning substantially narrows the gap, with a pipeline combining proxy data and manual captions achieving 69.4% average accuracy. The work highlights the value of data-centric approaches for enhancing fine-grained perception in multimodal video understanding and sets the stage for future integrations with architectural advances to handle longer sequences and more complex counting tasks.

Abstract

Fine-grained understanding of human actions and poses in videos is essential for human-centric AI applications. In this work, we introduce ActionArt, a fine-grained video-caption dataset designed to advance research in human-centric multimodal understanding. Our dataset comprises thousands of videos capturing a broad spectrum of human actions, human-object interactions, and diverse scenarios, each accompanied by detailed annotations that meticulously label every limb movement. We develop eight sub-tasks to evaluate the fine-grained understanding capabilities of existing large multimodal models across different dimensions. Experimental results indicate that, while current large multimodal models perform commendably on various tasks, they often fall short in achieving fine-grained understanding. We attribute this limitation to the scarcity of meticulously annotated data, which is both costly and difficult to scale manually. Since manual annotations are costly and hard to scale, we propose proxy tasks to enhance the model perception ability in both spatial and temporal dimensions. These proxy tasks are carefully crafted to be driven by data automatically generated from existing MLLMs, thereby reducing the reliance on costly manual labels. Experimental results show that the proposed proxy tasks significantly narrow the gap toward the performance achieved with manually annotated fine-grained data.

Figures (5)

• Figure 1: An example of the caption for consecutive frames in our ActionArt dataset. We provide detailed captions that describe human poses, body parts, and interactions with objects. The captions also convey temporal changes in the video at a fine-grained level.
• Figure 2: Our annotation pipeline starts by prompting GPT-4V to generate an initial caption for each video. We have observed that existing multimodal large language models (MLLMs) often struggle to accurately discern the directions of human body parts, resulting in coarse-grained descriptions, even with carefully designed prompts. Besides, the caption from GPT is noisy, and the errors within the captions are highlighted in red. After identifying these issues, we manually intervene and then prompt GPT-4 to generate various types of QA using corresponding templates, followed by manual refinement.
• Figure 3: Examples of different types QA.
• Figure 4: Automated data collection pipelines for various proxy tasks. For pose description and spatial difference mining, we meticulously design prompts to guide advanced MLLMs in reasoning about human poses and the spatial differences between two images. To obtain fine-grained QA, we avoid directly generating QA from long videos, as it is challenging for MLLMs to comprehend them at a detailed granularity. Instead, we use MLLMs to annotate pairwise frames as action units and synthesize long videos through concatenation.
• Figure 5: Comparison of direct QA generation from videos and our proposed pipeline. Directly generating QA from videos often lead to conflicts. As in the figure, both options C & D is correct for the question "When does the man put on the headphones?". As the granularity of the QA increases, conflicts are more likely to arise because existing MLLMs struggle to perceive all the fine-grained details in a video.