NoveltyBench: Evaluating Language Models for Humanlike Diversity

TL;DR

NoveltyBench introduces a diversity-focused benchmark for language-output generation, combining NB-Curated and NB-WildChat prompts with a partition-based notion of functional diversity ($distinct_k$) and a unified utility metric ($utility_k$) under a patience model. An equivalence classifier targets functionally distinct outputs, while a calibrated reward-based scoring pipeline anchors quality within a human-aligned utility framework. Empirical results across 20 frontier systems reveal that larger, more capable models often produce less diverse outputs, and that alignment procedures can further suppress distributional diversity despite quality gains. The work also demonstrates that explicit prompting strategies, notably in-context regeneration, can recover or exceed human-like diversity, underscoring the need for diversity-aware training and evaluation to improve user-perceived usefulness in generation tasks.

Abstract

Language models have demonstrated remarkable capabilities on standard benchmarks, yet they struggle increasingly from mode collapse, the inability to generate diverse and novel outputs. Our work introduces NoveltyBench, a benchmark specifically designed to evaluate the ability of language models to produce multiple distinct and high-quality outputs. NoveltyBench utilizes prompts curated to elicit diverse answers and filtered real-world user queries. Evaluating 20 leading language models, we find that current state-of-the-art systems generate significantly less diversity than human writers. Notably, larger models within a family often exhibit less diversity than their smaller counterparts, challenging the notion that capability on standard benchmarks translates directly to generative utility. While prompting strategies like in-context regeneration can elicit diversity, our findings highlight a fundamental lack of distributional diversity in current models, reducing their utility for users seeking varied responses and suggesting the need for new training and evaluation paradigms that prioritize diversity alongside quality.

Figures (8)

• Figure 1: NoveltyBench overview. We partition generations into functional equivalence classes and introduce two metrics: distinct$_k$ counts unique equivalence classes among $k$ samples, and utility$_k$ combines novelty and quality, weighing utility of individual generations by user patience ($p=0.8$) and only considering novel generations.
• Figure 2: Average number of unique generations out of a sample of 10 for all prompts in NoveltyBench. Error bars indicate 95% confidence intervals.
• Figure 3: Cumulative utility of generations of state-of-the-art models on NoveltyBench. A perfectly diverse and helpful model would have cumulative utility of 10. Error bars indicate 95% confidence intervals.
• Figure 4: As users demand more diverse generations, models become less useful. Larger models suffer more from degradation in utility.
• Figure 5: Alternative prompting methods can lead to improved novelty. The dashed lines report diversity and utility of answers handwritten by authors.