Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

A Stereotype Content Analysis on Color-related Social Bias in Large Vision Language Models

Junhyuk Choi, Minju Kim, Yeseon Hong, Bugeun Kim

TL;DR

This work tackles biases in large vision-language models by introducing SCM-based evaluation metrics and the BASIC color-aware benchmark to separate color effects from gender and race biases. It analyzes eight LVLMs across multiple architectures and sizes, revealing that color tone meaningfully shapes perceived competence and warmth, beyond what sentiment alone captures. The SCM-based approach proves more robust to safeguards that can distort polarity, and the findings show complex, non-uniform interactions between architecture, size, and bias. The results underscore the importance of including color-aware, semantically grounded metrics in auditing multimodal AI systems and point to future work on broader color palettes, causal model-architecture links, and multilingual analyses.

Abstract

As large vision language models(LVLMs) rapidly advance, concerns about their potential to learn and generate social biases and stereotypes are increasing. Previous studies on LVLM's stereotypes face two primary limitations: metrics that overlooked the importance of content words, and datasets that overlooked the effect of color. To address these limitations, this study introduces new evaluation metrics based on the Stereotype Content Model (SCM). We also propose BASIC, a benchmark for assessing gender, race, and color stereotypes. Using SCM metrics and BASIC, we conduct a study with eight LVLMs to discover stereotypes. As a result, we found three findings. (1) The SCM-based evaluation is effective in capturing stereotypes. (2) LVLMs exhibit color stereotypes in the output along with gender and race ones. (3) Interaction between model architecture and parameter sizes seems to affect stereotypes. We release BASIC publicly on [anonymized for review].

A Stereotype Content Analysis on Color-related Social Bias in Large Vision Language Models

TL;DR

This work tackles biases in large vision-language models by introducing SCM-based evaluation metrics and the BASIC color-aware benchmark to separate color effects from gender and race biases. It analyzes eight LVLMs across multiple architectures and sizes, revealing that color tone meaningfully shapes perceived competence and warmth, beyond what sentiment alone captures. The SCM-based approach proves more robust to safeguards that can distort polarity, and the findings show complex, non-uniform interactions between architecture, size, and bias. The results underscore the importance of including color-aware, semantically grounded metrics in auditing multimodal AI systems and point to future work on broader color palettes, causal model-architecture links, and multilingual analyses.

Abstract

As large vision language models(LVLMs) rapidly advance, concerns about their potential to learn and generate social biases and stereotypes are increasing. Previous studies on LVLM's stereotypes face two primary limitations: metrics that overlooked the importance of content words, and datasets that overlooked the effect of color. To address these limitations, this study introduces new evaluation metrics based on the Stereotype Content Model (SCM). We also propose BASIC, a benchmark for assessing gender, race, and color stereotypes. Using SCM metrics and BASIC, we conduct a study with eight LVLMs to discover stereotypes. As a result, we found three findings. (1) The SCM-based evaluation is effective in capturing stereotypes. (2) LVLMs exhibit color stereotypes in the output along with gender and race ones. (3) Interaction between model architecture and parameter sizes seems to affect stereotypes. We release BASIC publicly on [anonymized for review].

Paper Structure

This paper contains 41 sections, 5 equations, 2 figures, 29 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. BASIC Benchmark
  4. Step 1:
  5. Step 2:
  6. Step 3:
  7. Step 4:
  8. Step 5:
  9. Step 6:
  10. Experimental Setup
  11. Collecting Description
  12. Metrics
  13. Existing sentiment-based metric
  14. Our proposed SCM-based metrics
  15. Analyses
  16. ...and 26 more sections

Figures (2)

  • Figure 1: Examples from our BASIC dataset. Top row shows chess players with different colors (white, blue, red), retaining race and gender. Bottom row shows DJs with different race or gender, retaining white color.
  • Figure 2: Process of Image Generation in BASIC