HumaniBench: A Human-Centric Framework for Large Multimodal Models Evaluation

TL;DR

HumaniBench addresses the gap in evaluating large multimodal models (LMMs) on human-centered alignment by introducing a unified framework that maps seven HC principles to seven real-world, socially grounded tasks using about 32,000 expert-verified image-question pairs from news imagery. It combines a robust data collection pipeline, a HITL annotation process, and a shared evaluation suite to quantify fairness, ethics, understanding, reasoning, language inclusivity, empathy, and robustness across open-source and proprietary models. Key findings reveal consistent cross-model trade-offs: proprietary systems excel in ethics, reasoning, and empathy, while open-source models show strengths in visual grounding and robustness, yet none achieve comprehensive HC alignment, particularly in fairness and multilingual inclusivity. The work also demonstrates that techniques like chain-of-thought prompting and test-time scaling can improve HC dimensions by 8–12%, and it provides a reproducible foundation for systematic, multi-principle evaluation aimed at safer, more equitable real-world deployment.

Abstract

Although recent large multimodal models (LMMs) demonstrate impressive progress on vision language tasks, their alignment with human centered (HC) principles, such as fairness, ethics, inclusivity, empathy, and robustness; remains poorly understood. We present HumaniBench, a unified evaluation framework designed to characterize HC alignment across realistic, socially grounded visual contexts. HumaniBench contains 32,000 expert-verified image question pairs derived from real world news imagery and spanning seven evaluation tasks: scene understanding, instance identity, multiple-choice visual question answering (VQA), multilinguality, visual grounding, empathetic captioning, and image resilience testing. Each task is mapped to one or more HC principles through a principled operationalization of metrics covering accuracy, harmful content detection, hallucination and faithfulness, coherence, cross lingual quality, empathy, and robustness.We evaluate 15 state-of-the-art LMMs under this framework and observe consistent cross model trade offs: proprietary systems achieve the strongest performance on ethics, reasoning, and empathy, while open-source models exhibit superior visual grounding and resilience. All models, however, show persistent gaps in fairness and multilingual inclusivity. We further analyze the effect of inference-time techniques, finding that chain of thought prompting and test-time scaling yield 8 to 12 % improvements on several HC dimensions. HumaniBench provides a reproducible, extensible foundation for systematic HC evaluation of LMMs and enables fine-grained analysis of alignment trade-offs that are not captured by conventional multimodal benchmarks. https://vectorinstitute.github.io/humanibench/

Figures (10)

• Figure 1: HumaniBench Overview. The top panel illustrates our annotation pipeline that is rigorously performed by domain-expert verification. The mid panel presents 7 multimodal tasks (T1–T7) spanning both open- and closed-ended VQA. Each task maps to one or more human-aligned principles (center). The bottom panel depicts the evaluation workflow, with metrics.
• Figure 2: HumaniBench: Human-Centric AI Principles. The inner ring shows seven evaluation tasks (T1–T7); the middle ring lists the seven principles: Fairness, Ethics, Understanding, Reasoning, Language Inclusivity, Empathy, and Robustness. Thin dashed radial connectors make explicit that each task is evaluated under each principle, while the outer ring names broader societal governance pillars in HCAI.
• Figure 3: Semi-automated curation and annotation pipeline. Images are collected from news sites, deduplicated, annotated for captions and social attributes, and verified by experts.
• Figure 4: Comprehensive performance evaluation across tasks T1–T3. Columns correspond to T1 (Scene Understanding), T2 (Instance Identity), and T3 (Multiple-Choice VQA). Top row: radar charts compare models on four metrics (accuracy, faithfulness, contextual relevance, and coherence). Bottom row: representative benchmark examples with ground-truth answers and model responses.
• Figure 5: Performance breakdown of different LMMs across various tasks and social attributes.