GenColorBench: A Color Evaluation Benchmark for Text-to-Image Generation Models

TL;DR

The paper addresses the lack of fine-grained color controllability evaluation in text-to-image generation by introducing GenColorBench, a large-scale benchmark anchored in established color naming systems (ISCC-NBS and CSS3/X11) that also covers numerical color representations (RGB/HEX). It provides a five-task evaluation framework with perceptual-grounding and automated metrics to assess color fidelity in diverse model families, revealing persistent difficulties in precise color generation, common biases, and entanglement between color and object semantics. The work reports comprehensive benchmarking results across multiple state-of-the-art diffusion and unified models, identifies key failure modes (especially in numerical color understanding and multi-object color composition), and proposes baseline metrics and protocols to drive future improvements. By releasing the benchmark and methods publicly, the authors aim to catalyze advancements in color controllability for T2I systems and related design-oriented applications.

Abstract

Recent years have seen impressive advances in text-to-image generation, with image generative or unified models producing high-quality images from text. Yet these models still struggle with fine-grained color controllability, often failing to accurately match colors specified in text prompts. While existing benchmarks evaluate compositional reasoning and prompt adherence, none systematically assess color precision. Color is fundamental to human visual perception and communication, critical for applications from art to design workflows requiring brand consistency. However, current benchmarks either neglect color or rely on coarse assessments, missing key capabilities such as interpreting RGB values or aligning with human expectations. To this end, we propose GenColorBench, the first comprehensive benchmark for text-to-image color generation, grounded in color systems like ISCC-NBS and CSS3/X11, including numerical colors which are absent elsewhere. With 44K color-focused prompts covering 400+ colors, it reveals models' true capabilities via perceptual and automated assessments. Evaluations of popular text-to-image models using GenColorBench show performance variations, highlighting which color conventions models understand best and identifying failure modes. Our GenColorBench assessments will guide improvements in precise color generation. The benchmark will be made public upon acceptance.

Figures (12)

• Figure 1: An overview of GenColorBench evaluation framework. The evaluation pipeline consists of five key components: VQA-based object localization, object segmentation, pixel extraction, color grounding, and score mechanism. Then, five color evaluation tasks are devised to analyse different aspects of color understanding in T2I models covering single object coloring, color-object association, multi-object color composition, numerical color understanding, and Implicit Color Association.
• Figure 2: Performance of T2I model on category-wise color accuracy. The scores are averaged over the Level 2 and Level 3 ISCC-NBS colors, and CSS3/X11 colors based object focused prompts.
• Figure 3: Distribution of estimated dominant colors (Top-10) across 10,000 generated images for each T2I models, revealing inherent color biases in vanilla baseline models. Models include: A = PixArt Alpha, B = BLIP3o, F = Flux, J = Janus-Pro, N = Sana, O = OmniGen2, P = PixArt Sigma, S = Stable Diffusion 3, and D = Stable Diffusion 3.5. Interestingly, all the models are significantly biased towards black, gray, and brown across all the categories except fruits and vegetables.
• Figure 4: Color representation in LAION-2B text prompts, analyzed across four semantic categories: (i) Numeric Colors, (ii) ISCC-NBS L2 colors, (iii) CSS3/X11 named colors, and (iv) Color Modifiers. The data reveals the dominant representation of ISCC-NBS L2 colors and their modifiers. Whereas, the numeric colors are significantly under-represented as compared to the named colors.
• Figure 5: (Left) Comparison b/w basic and intermediate colors. These models better understand basic colors, while accuracy drops by 8--20% on intermediate colors. (Right) Comparison of color modifiers. These models understand light color modifiers better, while -ish modifiers remain worst.