HumanOmni: A Large Vision-Speech Language Model for Human-Centric Video Understanding

TL;DR

HumanOmni tackles human-centric video understanding by integrating vision and speech through a tri-branch visual architecture (face, body, interaction) fused via instruction-driven weighting, and an audio pathway aligned to text using Whisper and cross-modal training. It introduces a 2.4M video-clip dataset with 14M instructions, plus 50K emotion and 75K facial/speaker annotations, to pretrain and fine-tune specialized branches. The approach achieves state-of-the-art results on emotion recognition, facial description, and action understanding among open-source and omni models, with strong speech recognition performance and clear benefits from joint modalities. By open-sourcing the data and methods, the work advances practical, scalable human-centric audio-visual understanding and cross-modal reasoning.

Abstract

In human-centric scenes, the ability to simultaneously understand visual and auditory information is crucial. While recent omni models can process multiple modalities, they generally lack effectiveness in human-centric scenes due to the absence of large-scale, specialized datasets and non-targeted architectures. In this work, we developed HumanOmni, the industry's first human-centric Omni-multimodal large language model. We constructed a dataset containing over 2.4 million human-centric video clips with detailed captions and more than 14 million instructions, facilitating the understanding of diverse human-centric scenes. HumanOmni includes three specialized branches for understanding different types of scenes. It adaptively fuses features from these branches based on user instructions, significantly enhancing visual understanding in scenes centered around individuals. Moreover, HumanOmni integrates audio features to ensure a comprehensive understanding of environments and individuals. Our experiments validate HumanOmni's advanced capabilities in handling human-centric scenes across a variety of tasks, including emotion recognition, facial expression description, and action understanding. Our model will be open-sourced to facilitate further development and collaboration within both academia and industry.

Figures (4)

• Figure 1: Pipeline of HumanOmni. HumanOmni is a vision-speech language model that focus on human-centric scenes. For the visual component, we pre-trained three distinct branches using separate data. The features from these branches are fused based on user instructions. HumanOmni also supports audio input, enhancing its ability to fully understand complex human-centric scenes.
• Figure 2: Data processing flow. We employ scene detection and segmentation to divide the video into clips to prevent unnatural temporal changes caused by instantaneous scene transitions. Then, the clips with relatively low resolution are removed, and the key frames detection algorithms are applied, which helps to quantify the temporal changes in clips. To further improve learning efficiency, we generate brief captions based on advanced multimodal model, and eliminate the clips similar in contexts. Finally, in addition to being automatically annotated with human and face bounding boxes, the remaining video clips will be processed by several state-of-the-art multimodal models to generate detailed captions. Subsequently, a large language model will be used to synthesize the common content across these captions, while filtering out unique content that may result from model hallucinations.
• Figure 3: Instruction Data Generation Process for face-related, body-related, and interaction-related branches. We generate structured instruction data by leveraging Qwen2.5 with specifically designed prompts to process the detailed captions we have previously obtained.
• Figure 4: Illustration of the data annotation process. We annotate the data from the perspectives of Emotion, Speaker-specific speech, and Facial detailed description.