Beyond Divergent Creativity: A Human-Based Evaluation of Creativity in Large Language Models

TL;DR

This work argues that standard creativity benchmarks for large language models, exemplified by the Divergent Association Task (DAT), fail to capture the human creativity core of novelty plus appropriateness. It introduces Conditional Divergent Association Task (CDAT), a cue-conditioned framework that evaluates novelty conditional on contextual appropriateness, together with a Pareto-front analysis to characterize trade-offs. Across a broad set of models and temperatures, CDAT reveals systematic patterns: smaller, faster models tend to yield higher novelty but lower appropriateness, while larger, more aligned models favor appropriateness at the cost of novelty; conditioning consistently passes an appropriateness gate, validating novelty as a creativity signal within a contextual bound. The authors provide a dataset and code to support future work and propose CDAT as a practical, theory-grounded tool for robust creativity assessment in LLMs, complemented by diagnostic views to guide interpretation and model development.

Abstract

Large language models (LLMs) are increasingly used in verbal creative tasks. However, previous assessments of the creative capabilities of LLMs remain weakly grounded in human creativity theory and are thus hard to interpret. The widely used Divergent Association Task (DAT) focuses on novelty, ignoring appropriateness, a core component of creativity. We evaluate a range of state-of-the-art LLMs on DAT and show that their scores on the task are lower than those of two baselines that do not possess any creative abilities, undermining its validity for model evaluation. Grounded in human creativity theory, which defines creativity as the combination of novelty and appropriateness, we introduce Conditional Divergent Association Task (CDAT). CDAT evaluates novelty conditional on contextual appropriateness, separating noise from creativity better than DAT, while remaining simple and objective. Under CDAT, smaller model families often show the most creativity, whereas advanced families favor appropriateness at lower novelty. We hypothesize that training and alignment likely shift models along this frontier, making outputs more appropriate but less creative. We release the dataset and code.

Figures (15)

• Figure 1: Mean DAT scores across temperature settings with the DAT prompt, compared with two non-creative baselines: task-agnostic (horizontal dotted line) and task-aware (gray bars). Error bars denote 95% CIs for models; the shaded band shows the 95% CI for the task-agnostic baseline. All valid responses are included. Temperature settings: Low ($t{=}0.5$), Mid ($t{=}1.0$), High ($t{=}1.5$; not supported by all models). The task-aware baseline is evaluated at Mid only ($t{=}1.0$).
• Figure 2: CDAT appropriateness–novelty landscape (mid temperature). Solid markers show model means. The pink arrow indicates the ideal region under our operationalization (novelty evaluated after passing the appropriateness gate). The Pareto front (non-dominated models) traces optimal trade-offs. Circle A highlights basic models with high novelty and least appropriateness; Circle B marks advanced models with a balanced novelty-appropriateness trade-off.
• Figure 3: CDAT novelty distributions (mid temperature). Ridge plot of model novelty (DAT-style) under CDAT. Black vertical lines denote means. Detailed statistics and all temperatures are in Appendix \ref{['subsec:cdat-appdx-novelty']}.
• Figure 4: CDAT novelty–appropriateness landscape (middle temperature). Solid markers show model means; translucent markers show the first 30 per-cue points per model. The Pareto front (non-dominated models) traces optimal trade-offs. Means are jittered with Gaussian noise (mean $= 0$, SD $= 0.25$). All valid answers are used.
• Figure 5: CDAT novelty–appropriateness landscape (low temperature).