Deep Associations, High Creativity: A Simple yet Effective Metric for Evaluating Large Language Models

TL;DR

The paper presents PACE, a contamination-resistant framework to evaluate LLM creativity through Parallel Association Chains, using 110 IDS-derived seed words and three parallel 20-word chains per seed to compute an average association-distance score. It demonstrates strong alignment with human-centered creativity benchmarks, especially Arena Creative Writing ($\rho = 0.739$, $p < 0.001$), and shows that high-performing LLMs can match average human creativity but professionals remain superior, along with insights on concreteness and diversity of associations. The approach is scalable and data-efficient, leveraging automated scoring via embedding-based distances (e.g., FastText) and demonstrating robustness across model families and embedding choices. Overall, PACE offers a practical, low-cost tool for benchmarking LLM creativity, with implications for model development and evaluation, while acknowledging language and sample-size limitations.

Abstract

The evaluation of LLMs' creativity represents a crucial research domain, though challenges such as data contamination and costly human assessments often impede progress. Drawing inspiration from human creativity assessment, we propose PACE, asking LLMs to generate Parallel Association Chains to Evaluate their creativity. PACE minimizes the risk of data contamination and offers a straightforward, highly efficient evaluation, as evidenced by its strong correlation with Chatbot Arena Creative Writing rankings (Spearman's $ρ= 0.739$, $p < 0.001$) across various proprietary and open-source models. A comparative analysis of associative creativity between LLMs and humans reveals that while high-performing LLMs achieve scores comparable to average human performance, professional humans consistently outperform LLMs. Furthermore, linguistic analysis reveals that both humans and LLMs exhibit a trend of decreasing concreteness in their associations, and humans demonstrating a greater diversity of associative patterns.

Figures (8)

• Figure 1: PACE evaluation process: For each seed word, three 20-word association chains are generated and their association distances are averaged to obtain the seed score. The model's creativity score is calculated by averaging all seed scores.
• Figure 2: Comparison of model rankings according to PACE and Arena Creative Writing. Each point represents a language model, where different release versions of the same model are treated as separate variants, with the x-axis showing the PACE rank (based on association distance) and the y-axis showing the Arena Creative Writing rank. The red dashed line indicates the Spearman rank correlation fit ($\rho = 0.739$, $p < 0.001$). Claude-3.5-Sonnet achieves the highest PACE ranking among the evaluated models.
• Figure 3: Comparison of association distances between humans and LLMs. Using human data from gray2019forward, results show that high-performing LLMs match average human performance, but fall short of professional humans.
• Figure 4: Average concreteness scores across chain positions for different groups. All groups show declining concreteness, with models exhibiting higher concreteness than humans.
• Figure 5: Types of associations within chains, categorized according to the association type framework described by nissen2006word. Details are provided in Appendix \ref{['sec:prediction_of_concreteness_using_embedding_models']}.