CreativityPrism: A Holistic Benchmark for Large Language Model Creativity

TL;DR

CreativityPrism addresses the challenge of evaluating LLM creativity in a scalable, holistic manner. It decomposes creativity into quality, novelty, and diversity and tests nine tasks across three domains—divergent thinking, creative writing, and logical reasoning—using 20 metrics. By aggregating task-specific metrics into dimension scores and an overall creativity score, the framework reveals systematic differences between proprietary and open models and uncovers domain-specific correlations among creativity dimensions. The results argue for multi-task, multi-dimension evaluation to accurately assess LLM creativity and guide future model development. Limitations include language scope, evaluator bias from LLM-based judgments, and a focus on text-only tasks, suggesting avenues for future multimodal and fine-tuning studies.

Abstract

Creativity is often seen as a hallmark of human intelligence. While large language models (LLMs) are increasingly perceived as producing creative text, there is still no holistic framework to evaluate their creativity across diverse scenarios. Existing evaluation methods remain fragmented, with dramatic variation across domains and tasks, largely due to differing definitions and measurements of creativity. Inspired by the hypothesis that creativity is not one fixed idea, we propose CreativityPrism, an evaluation analysis framework that decomposes creativity into three dimensions: quality, novelty, and diversity. CreativityPrism incorporates nine tasks, three domains, i.e., divergent thinking, creative writing, and logical reasoning, and twenty evaluation metrics, which measure each dimension in task-specific, unique ways. We evaluate 17 state-of-the-art (SoTA) proprietary and open-sourced LLMs on CreativityPrism and analyze the performance correlations among different metrics and task domains. Our results reveal a notable gap between proprietary and open-source models. Overall, model performance tends to be highly correlated across tasks within the same domain and less so across different domains. Among evaluation dimensions, diversity and quality metrics show strong correlations - models that perform well on one often excel on the other - whereas novelty exhibits much weaker correlation with either. These findings support our hypothesis that strong performance in one creativity task or dimension does not necessarily generalize to others, underscoring the need for a holistic evaluation of LLM creativity.

Figures (8)

• Figure 1: Overview of CreativityPrism. We evaluated 17 LLMs across nine datasets spanning three domains: divergent thinking, creative writing, and logical reasoning. Each LLM was prompted to complete the tasks, and their outputs were evaluated using task-specific metrics. However, these metrics are diverse and difficult to interpret holistically in terms of machine creativity. To address this, we organize the metrics into three key dimensions of creativity: quality, novelty, and diversity. Creativity cannot be captured by a single measure—it must be evaluated through multiple dimensions. Task details can be found in Table \ref{['tab:task-descriptions']}.
• Figure 2: (a) Performance v.s. Day since LLM release date. The line represents best fit linear regression. We can see that model performance in all dimensions has seen improvements over time. (b) Performance gap between the open-sourced models and the proprietary models, averaged by model size group.
• Figure 3: Models' performance correlations, grouped by task and domain; C-Index refers to Creativity Index; C-Short refers to Creative Short Story; C-Math refers to Creative Math; all correlations are Pearson's correlation.
• Figure 4: Left: models' performance correlations, grouped by creativity dimensions; Right: individual model performance, min-max normalized by domains. TTCT and TTCW tasks are omitted here as they have very high inter-task correlation. A full version of the heat map can be found in Figure \ref{['fig:heatmap-full']}.
• Figure 5: Overall performances.