ColorConceptBench: A Benchmark for Probabilistic Color-Concept Understanding in Text-to-Image Models

TL;DR

ColorConceptBench introduces a human-grounded benchmark to evaluate probabilistic color–concept understanding in text-to-image models by collecting 6,369 colorizations across 1,281 implicit concepts and extracting human color distributions $P_H(x|c)$. The authors couple concept selection, human colorization, and a probabilistic color extraction pipeline to compare model outputs $P_M(x|c)$ using metrics such as $EMD$, $PCC$, and $ED$, plus deterministic cues like $DCA$ and $\Delta\text{Hue}$. Across seven open-source T2I models and varied styles and CFG scales, results show only moderate alignment with human color semantics, with abstract concepts proving particularly challenging and scaling or stronger guidance failing to close the gap; $EMD$ shows the strongest correspondence with human judgments. The work highlights the need for fundamental advances in how models learn and represent implicit meaning, provides a rigorous evaluation framework, and offers a dataset and protocol to guide future research in semantically aware color generation.

Abstract

While text-to-image (T2I) models have advanced considerably, their capability to associate colors with implicit concepts remains underexplored. To address the gap, we introduce ColorConceptBench, a new human-annotated benchmark to systematically evaluate color-concept associations through the lens of probabilistic color distributions. ColorConceptBench moves beyond explicit color names or codes by probing how models translate 1,281 implicit color concepts using a foundation of 6,369 human annotations. Our evaluation of seven leading T2I models reveals that current models lack sensitivity to abstract semantics, and crucially, this limitation appears resistant to standard interventions (e.g., scaling and guidance). This demonstrates that achieving human-like color semantics requires more than larger models, but demands a fundamental shift in how models learn and represent implicit meaning.

Figures (14)

• Figure 1: Unlike explicit color matching (top), ColorConceptBench evaluates implicit semantic alignment using probabilistic color distributions (bottom).
• Figure 2: Dataset Statistics and Construction Pipeline. An overview of the hierarchical concept distribution and our three-stage construction process: concept selection, human-grounded colorization, and probabilistic color extraction.
• Figure 3: Qualitative comparison of color-concept association across different text-to-image models. Colors shift for base nouns (e.g., 'cabin') and modified concepts involving visual states (e.g., 'cozy') or emotions (e.g., 'lonely'), across both natural and clipart styles, shown with color distribution and dominant colors with sample number.
• Figure 4: Models consistently exhibit lower color shift magnitudes than the human baseline (left), conservatively prioritizing intrinsic object colors over modifier-induced adjustments (right).
• Figure 5: Impact of Guidance Scale. Increasing the CFG scale generally leads to higher EMD and lower PCC across models (worse alignment).