Has the Creativity of Large-Language Models peaked? An analysis of inter- and intra-LLM variability

TL;DR

The paper interrogates whether large language models have become more creative over time and how consistent their creative outputs are, using two standard DT measures (DAT and AUT) across 14 diverse LLMs. It reveals no clear evidence of widespread creative gains since 2023, with notable inter-model differences and substantial intra-model variability; even highly capable models seldom produce outputs in the top decile of human creativity. AUT outputs generally exceed average human performance on average, but only a tiny fraction reach top-human levels, and prompting effects can shift results significantly. The findings underscore the need for nuanced evaluation frameworks, careful model and prompt selection, and repeated assessments in human-AI co-creative workflows to avoid overstating LLMs’ creative potential. Overall, while GenAI can boost ideation speed and output fluency, it tends to favor combinatorial rather than radical creativity, reinforcing the importance of thoughtful human oversight and framing in creative tasks.

Abstract

Following the widespread adoption of ChatGPT in early 2023, numerous studies reported that large language models (LLMs) can match or even surpass human performance in creative tasks. However, it remains unclear whether LLMs have become more creative over time, and how consistent their creative output is. In this study, we evaluated 14 widely used LLMs -- including GPT-4, Claude, Llama, Grok, Mistral, and DeepSeek -- across two validated creativity assessments: the Divergent Association Task (DAT) and the Alternative Uses Task (AUT). Contrary to expectations, we found no evidence of increased creative performance over the past 18-24 months, with GPT-4 performing worse than in previous studies. For the more widely used AUT, all models performed on average better than the average human, with GPT-4o and o3-mini performing best. However, only 0.28% of LLM-generated responses reached the top 10% of human creativity benchmarks. Beyond inter-model differences, we document substantial intra-model variability: the same LLM, given the same prompt, can produce outputs ranging from below-average to original. This variability has important implications for both creativity research and practical applications. Ignoring such variability risks misjudging the creative potential of LLMs, either inflating or underestimating their capabilities. The choice of prompts affected LLMs differently. Our findings underscore the need for more nuanced evaluation frameworks and highlight the importance of model selection, prompt design, and repeated assessment when using Generative AI (GenAI) tools in creative contexts.

Figures (12)

• Figure 1: Percentile scores of each large language model (LLM) in the DAT-awareness condition. The first group reflects the distribution of human percentile ranks. Percentiles for the 14 LLMs were computed by benchmarking their DAT scores against the human distribution, such that higher percentiles indicate better performance relative to humans Note. Each boxplot displays the distribution of percentiles for a given LLM, with black diamonds indicating mean performance. The red dashed line represents the average human performance (50th percentile). The human responses are from olson_naming_2021
• Figure 2: Percentile scores of each large language model (LLM) in the creative-prompt condition (AUT). The first group reflects the distribution of human percentile ranks. Percentiles for the 14 LLMs were computed by benchmarking their AUT scores against the human distribution, such that higher percentiles indicate better performance relative to humans Note. Each boxplot displays the distribution of percentiles for a given LLM, with black diamonds indicating mean performance. The red dashed line represents the average human performance (50th percentile). The human responses are from hubert_current_2024
• Figure 3: Percentile scores of each large language model (LLM) in the DAT-awareness. Significant pairwise comparisons between any two groups are displayed. The first group reflects the distribution of human scores.
• Figure 4: Scores of each large language model (LLM) in the DAT-awareness condition. The first group reflects the distribution of human scores.
• Figure 5: Scores of each large language model (LLM) in the DAT-awareness condition. Significant pairwise comparisons between any two groups are displayed. The first group reflects the distribution of human scores.