Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Rethinking Ground Truth: A Case Study on Human Label Variation in MLLM Benchmarking

Tomas Ruiz, Tanalp Agustoslu, Carsten Schwemmer

Abstract

Human Label Variation (HLV), i.e. systematic differences among annotators' judgments, remains underexplored in benchmarks despite rapid progress in large language model (LLM) development. We address this gap by introducing an evaluation protocol for multimodal large language model (MLLM) benchmarking that explicitly accounts for two conditions: (1) human label agreement and (2) disagreement. We apply this protocol to two state-of-the-art MLLM families (Gemma 3, Qwen 2.5 VL) using non-aggregated human annotations from a social media content classification dataset. Across tasks, we find that larger models tend to perform best on high-agreement subsets, yet often underperform medium-sized models when human disagreement is high, indicating that parameter count alone does not determine sensitivity to ambiguity and subjectivity. These results show that benchmarks based solely on consensus labels can overstate model capabilities in such domains and that incorporating human label variation yields more realistic and robust assessments of MLLMs in content moderation pipelines.

Rethinking Ground Truth: A Case Study on Human Label Variation in MLLM Benchmarking

Abstract

Human Label Variation (HLV), i.e. systematic differences among annotators' judgments, remains underexplored in benchmarks despite rapid progress in large language model (LLM) development. We address this gap by introducing an evaluation protocol for multimodal large language model (MLLM) benchmarking that explicitly accounts for two conditions: (1) human label agreement and (2) disagreement. We apply this protocol to two state-of-the-art MLLM families (Gemma 3, Qwen 2.5 VL) using non-aggregated human annotations from a social media content classification dataset. Across tasks, we find that larger models tend to perform best on high-agreement subsets, yet often underperform medium-sized models when human disagreement is high, indicating that parameter count alone does not determine sensitivity to ambiguity and subjectivity. These results show that benchmarks based solely on consensus labels can overstate model capabilities in such domains and that incorporating human label variation yields more realistic and robust assessments of MLLMs in content moderation pipelines.
Paper Structure (21 sections, 2 equations, 1 figure, 3 tables)

This paper contains 21 sections, 2 equations, 1 figure, 3 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Benchmark Datasets on Content Moderation
  4. Human Label Variation
  5. Methods
  6. Dataset
  7. Evaluation Protocol
  8. Results
  9. Agreement subset
  10. Disagreement subset
  11. Discussion
  12. Instruction Tuning and Alignment Tax
  13. Shortcut Reasoning and Multimodal Understanding
  14. Limitations and Future Work
  15. Conclusion
  16. ...and 6 more sections

Figures (1)

  • Figure 1: (Top) Correlation of runtime, output tokens generated and input frames per video. (Bottom) GPU memory usage across models. All models fit on a single H100 (96GB) except Qwen2.5-VL 72B, which required 3× H100 GPUs. We use HuggingFace to run our experiments. Total compute is approximately 80 GPU hours.