HumanVLM: Foundation for Human-Scene Vision-Language Model

TL;DR

A domain-specific Large Vision-Language Model, Human-Scene Vision-Language Model (HumanVLM), designed to provide a foundation for human-scene Vision-Language tasks, and will stimulate the research in human-around fields is introduced.

Abstract

Human-scene vision-language tasks are increasingly prevalent in diverse social applications, yet recent advancements predominantly rely on models specifically tailored to individual tasks. Emerging research indicates that large vision-language models (VLMs) can enhance performance across various downstream vision-language understanding tasks. However, general-domain models often underperform in specialized fields. This study introduces a domain-specific Large Vision-Language Model, Human-Scene Vision-Language Model (HumanVLM), designed to provide a foundation for human-scene Vision-Language tasks. Specifically, (1) we create a large-scale human-scene multimodal image-text dataset (HumanCaption-10M) sourced from the Internet to facilitate domain-specific alignment; (2) develop a captioning approach for human-centered images, capturing human faces, bodies, and backgrounds, and construct a high-quality Human-Scene image-text dataset (HumanCaptionHQ, about 311k pairs) that contain as much detailed information as possible about human; (3) Using HumanCaption-10M and HumanCaptionHQ, we train a HumanVLM. In the experiments, we then evaluate our HumanVLM across varous downstream tasks, where it demonstrates superior overall performance among multimodal models of comparable scale, particularly excelling in human-related tasks and significantly outperforming similar models, including Qwen2VL and ChatGPT-4o. HumanVLM, alongside the data introduced, will stimulate the research in human-around fields.

Figures (11)

• Figure 1: Comparison with various VLMs. Our HumanVLM achieves the best overall performance on a broad range of tasks compared with other generalist models.
• Figure 2: Illustrations of our HumanVLM. It is capable of answering various questions based on the Human-Scene image.
• Figure 3: Pipelines of generating the caption for human-scene image.
• Figure 4: we calculated the proportions of different scores within each dataset. Lower (High) scores of BRISQUE (CLIPIQA) indicate better image quality.
• Figure 5: Text distribution. (a) Cumulative proportion of sentences with varying word counts in each dataset. (b) Unique 4-grams count by percentage.