Death of the Novel(ty): Beyond n-Gram Novelty as a Metric for Textual Creativity

TL;DR

It is found that while n-gram novelty is positively associated with expert writer-judged creativity, approximately 91% of top-quartile n-gram novel expressions are not judged as creative, cautioning against relying on n-gram novelty alone.

Abstract

N-gram novelty is widely used to evaluate language models' ability to generate text outside of their training data. More recently, it has also been adopted as a metric for measuring textual creativity. However, theoretical work on creativity suggests that this approach may be inadequate, as it does not account for creativity's dual nature: novelty (how original the text is) and appropriateness (how sensical and pragmatic it is). We investigate the relationship between this notion of creativity and n-gram novelty through 8,618 expert writer annotations of novelty, pragmaticality, and sensicality via close reading of human- and AI-generated text. We find that while n-gram novelty is positively associated with expert writer-judged creativity, approximately 91% of top-quartile n-gram novel expressions are not judged as creative, cautioning against relying on n-gram novelty alone. Furthermore, unlike in human-written text, higher n-gram novelty in open-source LLMs correlates with lower pragmaticality. In an exploratory study with frontier closed-source models, we additionally confirm that they are less likely to produce creative expressions than humans. Using our dataset, we test whether zero-shot, few-shot, and finetuned models are able to identify expressions perceived as novel by experts (a positive aspect of writing) or non-pragmatic (a negative aspect). Overall, frontier LLMs exhibit performance much higher than random but leave room for improvement, especially struggling to identify non-pragmatic expressions. We further find that LLM-as-a-Judge novelty ratings align with expert writer preferences in an out-of-distribution dataset, more so than an n-gram based metric.

Figures (11)

• Figure 1: Examples of non-pragmatic expressions (top), and expressions that are both non-sensical and non-pragmatic (bottom).
• Figure 2: Example of the annotation interface and an expert writer's annotation.
• Figure 3: Predicted probability of being rated as pragmatic for different values of log-std perplexity, by generation source. Bands indicate 95% CIs. Annotator and paragraph intercepts correspond to population-level fixed effects.
• Figure 4: Comparison of finetuned, zero-shot and few-shot model performance across (a) perceived novel and (b) non-pragmatic expression identification tasks (F1 scores with 95% CIs).
• Figure 5: Comparison of how predictive LLM-J novelty ($\Delta \text{Nov}_{AB}$) and pragmaticality ($\Delta \text{Prag}_{AB}$) score differences are of expert vs. crowd preferences, bands indicate 95% confidence intervals.