A Confederacy of Models: a Comprehensive Evaluation of LLMs on Creative Writing

TL;DR

Problem: evaluating LLMs on creative writing is challenging due to long-range coherence, voice, and humor. Approach: a pure zero-shot, single-prompt evaluation comparing 12 instruction-aligned LLMs against five human writers using a 10-item rubric, with 60 AI-generated and 5 human stories rated by 10 experts. Findings: GPT-4 and Claude lead across many criteria; humans excel in originality and humor; open-source models lag; some LLMs rival humans on readability and epicness, though humor remains hard. Significance: the study demonstrates that current commercial LLMs can produce high-quality creative writing in a zero-shot setting and highlights evaluation design considerations, limitations, and directions for future cross-genre and cross-language research.

Abstract

We evaluate a range of recent LLMs on English creative writing, a challenging and complex task that requires imagination, coherence, and style. We use a difficult, open-ended scenario chosen to avoid training data reuse: an epic narration of a single combat between Ignatius J. Reilly, the protagonist of the Pulitzer Prize-winning novel A Confederacy of Dunces (1980), and a pterodactyl, a prehistoric flying reptile. We ask several LLMs and humans to write such a story and conduct a human evalution involving various criteria such as fluency, coherence, originality, humor, and style. Our results show that some state-of-the-art commercial LLMs match or slightly outperform our writers in most dimensions; whereas open-source LLMs lag behind. Humans retain an edge in creativity, while humor shows a binary divide between LLMs that can handle it comparably to humans and those that fail at it. We discuss the implications and limitations of our study and suggest directions for future research.

Figures (14)

• Figure 1: Box plot comparing overall ratings for stories by humans and 12 LLMs, arranged left to right by mean overall rating. Boxes show median, quartiles Q1-Q3, and whiskers at 1.5 IQR, with values outside that range plotted as outliers. Filled red circles represent means.
• Figure 2: Box plot comparing humor ratings for stories generated by humans and 12 LLMs, sorted left to right by mean overall rating. Notation as in Figure \ref{['fig:box_overall']}.
• Figure 3: Box plot comparing creativity ratings for stories generated by humans and 12 LLMs, sorted left to right by mean overall rating. Notation as in Figure \ref{['fig:box_overall']}.
• Figure 4: Box plot comparing epicness ratings for stories generated by humans and 12 LLMs, sorted left to right by mean overall rating. Notation as in Figure \ref{['fig:box_overall']}.
• Figure 5: Box plot comparing rubric item 1 (cohesion) for stories generated by humans and 12 LLMs, sorted left to right by mean overall rating. Notation as in Figure \ref{['fig:box_overall']}.