We're Different, We're the Same: Creative Homogeneity Across LLMs

TL;DR

This paper investigates whether creative outputs homogenize across a broad array of LLMs or are model-specific. Using standardized creativity tests (AUT, FF, DAT), it compares originality $\mathcal{O}_t(\mathcal{P})$ and population variability $\mathcal{V}_t(\mathcal{P})$ between 22 LLMs and humans, finding strong cross-LLM homogeneity despite comparable individual creativity. The effect persists after controlling for output structure and model family, though prompt engineering can modestly increase variability. These results imply that widespread use of LLMs as creative partners may steer users toward a narrow, shared set of ideas, with implications for cultural diversity and epistemic resilience.

Abstract

Numerous powerful large language models (LLMs) are now available for use as writing support tools, idea generators, and beyond. Although these LLMs are marketed as helpful creative assistants, several works have shown that using an LLM as a creative partner results in a narrower set of creative outputs. However, these studies only consider the effects of interacting with a single LLM, begging the question of whether such narrowed creativity stems from using a particular LLM -- which arguably has a limited range of outputs -- or from using LLMs in general as creative assistants. To study this question, we elicit creative responses from humans and a broad set of LLMs using standardized creativity tests and compare the population-level diversity of responses. We find that LLM responses are much more similar to other LLM responses than human responses are to each other, even after controlling for response structure and other key variables. This finding of significant homogeneity in creative outputs across the LLMs we evaluate adds a new dimension to the ongoing conversation about creativity and LLMs. If today's LLMs behave similarly, using them as a creative partners -- regardless of the model used -- may drive all users towards a limited set of "creative" outputs.

Figures (10)

• Figure 1: LLMs slightly outperform humans on the AUT and DAT, but humans slightly outperform LLMs on FF.
• Figure 2: LLM responses exhibit far less variability than human responses, as measured by cosine distance between embedded responses.
• Figure 3: LLM responses cluster together in feature space more than do human responses. K-means clustering of TSNE of AUT sentence embeddings.
• Figure 4: LLM responses have far more words in common than do human responses. We look at word overlaps between "full" responses from LLMs and humans— e.g. all uses from the AUT, all words in the FF, etc. This corresponds to the sentence embedding method of population originality measurement, and explains why the difference between LLMs and humans is more pronounced in this setting.
• Figure 5: Effect of different AUT prompt wordings on length of LLM AUT responses. We use prompt verison 3 in most experiments in this paper, since LLM responses to this prompt most closely match the human distribution of response lengths.