GenColor: Generative Color-Concept Association in Visual Design

TL;DR

This work introduces GenColor, a diffusion-model–driven framework for color-concept association in visual design that overcomes the instability and context limitations of query-based color references. By generating context-aware concept images, segmenting concept-relevant regions with an open-set segmentation ensemble, and extracting robust primary-accent color palettes, GenColor provides designers with justified, objective color references and flexible palettes across diverse contexts. A formative study with professional designers informs design goals, while a large designer-built coloring dataset and quantitative/user evaluations demonstrate that GenColor often produces colors closer to human judgments than query-based methods, particularly for context-dependent concepts. The framework is complemented by practical applications (design-element coloring, clipart coloring) and a public gallery for retrieval, offering a scalable, designer-aligned tool to support color decision-making in real-world design tasks.

Abstract

Existing approaches for color-concept association typically rely on query-based image referencing, and color extraction from image references. However, these approaches are effective only for common concepts, and are vulnerable to unstable image referencing and varying image conditions. Our formative study with designers underscores the need for primary-accent color compositions and context-dependent colors (e.g., 'clear' vs. 'polluted' sky) in design. In response, we introduce a generative approach for mining semantically resonant colors leveraging images generated by text-to-image models. Our insight is that contemporary text-to-image models can resemble visual patterns from large-scale real-world data. The framework comprises three stages: concept instancing produces generative samples using diffusion models, text-guided image segmentation identifies concept-relevant regions within the image, and color association extracts primarily accompanied by accent colors. Quantitative comparisons with expert designs validate our approach's effectiveness, and we demonstrate the applicability through cases in various design scenarios and a gallery.

Figures (16)

• Figure 1: Design scenarios where appropriate color-concept associations are effective to enhance visual communication: (A) graphic design and data visualization. These scenarios require color-concept association, which is preferable in generated images. When comparing the queried and generated images for the "Statue of Liberty", (B) colors extracted from the queried images (both photos and clipart) show significant variations, whereas (C) the generated images provide a more consistent color representation.
• Figure 2: Usage scenario for designing a clipart to raise awareness about mountain pollution. (a) Illustration of the design elements, including the concept ("mountain") and the context ("pristine"vs."pollution"). (b) Examples of clipart design: the clipart for "pristine mountain" uses bright green, whilst the one for "polluted mountain" uses grayish green. (c) Primary-accent color composition is used by designers, with the primary color representing the concept and the context, while accent colors provide depth and decoration.
• Figure 3: Overview of the GenColor framework. The framework includes three stages: Concept Instancing for generating representative image samples, Text-guided Image Segmentation for identifying relevant regions, and Color Association for extracting the primary-accent color composition.
• Figure 4: Prompt design and parameter setting in the Conceptual Instancing stage. The prompt is refined based on the core principles of concept, context, style, lighting, and quality control. The guidance scale and seed are adjusted to control image quality and diversity.
• Figure 5: The text-guided image segmentation process. (a) The pipeline utilizes G-DINO for prompt-guided detection and SAM for concept-based segmentation. (b) Comparison of traditional background removal and text-guided segmentation.