Large Language Models estimate fine-grained human color-concept associations

TL;DR

This study suggests that high-order covariances between language and perception, as expressed in the natural environment of the internet, contain sufficient information to support learning of human-like color-concept associations, and provides an existence proof that a learning system can encode such associations without initial constraints.

Abstract

Concepts, both abstract and concrete, elicit a distribution of association strengths across perceptual color space, which influence aspects of visual cognition ranging from object recognition to interpretation of information visualizations. While prior work has hypothesized that color-concept associations may be learned from the cross-modal statistical structure of experience, it has been unclear whether natural environments possess such structure or, if so, whether learning systems are capable of discovering and exploiting it without strong prior constraints. We addressed these questions by investigating the ability of GPT-4, a multimodal large language model, to estimate human-like color-concept associations without any additional training. Starting with human color-concept association ratings for 71 color set spanning perceptual color space (\texttt{UW-71}) and concepts that varied in abstractness, we assessed how well association ratings generated by GPT-4 could predict human ratings. GPT-4 ratings were correlated with human ratings, with performance comparable to state-of-the-art methods for automatically estimating color-concept associations from images. Variability in GPT-4's performance across concepts could be explained by specificity of the concept's color-concept association distribution. This study suggests that high-order covariances between language and perception, as expressed in the natural environment of the internet, contain sufficient information to support learning of human-like color-concept associations, and provides an existence proof that a learning system can encode such associations without initial constraints. The work further shows that GPT-4 can be used to efficiently estimate distributions of color associations for a broad range of concepts, potentially serving as a critical tool for designing effective and intuitive information visualizations.

Figures (22)

• Figure 1: Colors in the UW--71 color library plotted in CIELAB space (figure adapted from Mukherjee et al. mukherjee2021context.
• Figure 2: Example trial from the human color-concept association rating task (left) and GPT-4 rating task for the same color-concept pair (right). Bar graphs correspond to average human and GPT-4 color concept association ratings for the concept 'bird' over the entire UW-71 color library.
• Figure 3: The relationship between specificity and concreteness for the set of 70 concepts. The color-concept association distributions for each of the concepts bird, carrot, leisure, and love are plotted on a scale from 0 to 1.
• Figure 4: Correlations between average human color-concept association ratings and predicted ratings from GPT-4 from experiments 1 (light grey bars), 2 (medium gray bars), and 3 (dark grey bars) for each of our 70 concepts. Teal lines above each concept correspond to the average human split-half reliability for that concept.
• Figure 5: (A) Relationship between the specificity of concepts' color-concept associations and GPT-4's ability to accurately predict associations. Each point represents a different concept. (B) Relationship between the concepts' concreteness and GPT-4's ability to accurately predict associations. The first column in (A) and (B) reflect human split-half correlations for color-concept associations as a function of specificity (top) and concreteness (bottom).