Measuring LLM Novelty As The Frontier Of Original And High-Quality Output

TL;DR

The paper defines novelty as the harmonic mean of output originality, measured by the fraction of unseen $n$-grams relative to training data, and task-specific quality. Using open-data model families (OLMo, OLMo-2, Pythia) across story completion, poetry, and MacGyver-style tasks, it shows that model scaling and post-training consistently improve novelty by boosting quality, while stronger base models increase originality. Inference-time prompts yield limited gains and often reduce quality, underscoring tradeoffs in elicitation strategies. The authors release a dataset of thousands of generations and advocate using the joint novelty metric to track generalization and creativity across evolving models and applications. The work also discusses extensions to black-box models where aggregated novelty scores can be reported without exposing proprietary data, supporting AI safety and transparency in creativity-enabled AI systems.

Abstract

As large language models (LLMs) are increasingly used for ideation and scientific discovery, it is important to evaluate their ability to generate novel output. Prior work evaluates novelty as originality with respect to model training data, but original outputs may be of low quality. In contrast, non-expert judges more reliably score quality but may favor memorized outputs, limiting the reliability of human preference as a metric. We introduce a new novelty metric for LLM generations that balances originality and quality -- the harmonic mean of the fraction of \ngrams unseen during training and a task-specific quality score. Using this framework, we identify trends that affect the novelty of generations from three families of open-data models (OLMo, OLMo-2, and Pythia) on three creative tasks: story completion, poetry writing, and creative tool use. We find that model-generated text from some base LLMs is less novel than human-written text from the internet. However, increasing model scale and post-training reliably improves novelty due to improvements in output quality. We also find that improving the base model at the same scale (\eg OLMo 7B to OLMo-2 7B) leads to higher novelty due to higher originality. Finally, we observe that inference-time methods, such as prompting and providing novel in-context examples, have a much smaller effect on novelty, often increasing originality at the expense of quality. This highlights the need for further research into more effective elicitation strategies as we use models for creative applications.

Figures (7)

• Figure 1: We evaluate LLMs’ ability to generate novel text, defined as high-quality responses that avoid reproducing higher-order $n$-grams from training data (highlighted in blue). Novelty is measured as the harmonic mean of unseen $n$-gram fraction ($x$-axis) and output quality ($y$-axis) (\ref{['sec:formulation']}). Contour lines denote equal novelty in each plot. We find that: (a) scaling models and (b) post-training increase novelty through improved quality, while (c) stronger base models (e.g., OLMo 1 to OLMo 2) improve novelty by generating more original output (\ref{['sec:results_base']}). Inference-time methods (e.g., novel ICL examples, Denial Prompting) have limited effect on shifting the novelty frontier (\ref{['sec:results_elicit']}).
• Figure 2: Comparing novelty of base and post-trained LLMs by plotting output quality ($y$-axis) vs $n$-gram originality for $n=5$ ($x$-axis) for CoPoet, TinyStories and MacGyver. Post-training uniformly increases novelty at all model sizes for both OLMo and OLMo-2.
• Figure 3: Comparing novelty of models by plotting output quality ($y$-axis) vs $n$-gram originality for $n=5$ ($x$-axis) for CoPoet, TinyStories and MacGyver. Improving the underlying base LLM (OLMo to OLMo-2 and Pythia to Pythia-DDP) leads to higher novelty at the same model scale for all tasks, driven by higher originality. Increasing model scale (darker colors) leads to higher novelty driven by higher output quality, particularly on TinyStories and MacGyver.
• Figure 4: Effect of varying sampling temperature on novelty by plotting output quality ($y$-axis) vs $n$-gram originality for $n=5$ ($x$-axis) for CoPoet, TinyStories, and MacGyver. Increasing sampling temperature (darker colors) from $0.5$ to $2$ for OLMo-7B and Pythia-12B increases originality, with a cost to output quality, resulting in similar novelty levels (\ref{['sec:results_elicit_output']}). \ref{['tab:results_elicit_temp']} has the raw scores.
• Figure 5: Effect of varying the prompting method on novelty by plotting output quality ($y$-axis) vs $n$-gram originality for $n=5$ ($x$-axis) for CoPoet, TinyStories, and MacGyver. Different prompting methods---providing novel ICL examples (\ref{['sec:results_elicit_input_dist']}) for Base models, and Asking for novelty and Denial Prompting on Instruct models (\ref{['sec:results_elicit_input_prompt']})---have little effect on novelty, and often trade off a small increase of originality for slightly lower quality. \ref{['fig:pareto_frontier_appendix_all_n']} shows the same plot for other $n$.