Confidence, Not Perplexity: A Better Metric for the Creative Era of LLMs

TL;DR

The paper tackles the bias of reference-free evaluation toward safe, high-frequency language in creative LLM outputs by introducing Confidence Score (CS), a distribution-shape metric computed from the output probabilities. CS combines the probability of the chosen token with the dispersion among top candidates, using Avg CS and Worst-Case CS to synthesize sequence-level confidence. In experiments with gpt-4o-mini across creative prompts and categorized difficulty prompts, CS shows markedly reduced creativity bias compared with self-perplexity and fluency measures while remaining effective at distinguishing task difficulty. The findings suggest CS offers a more balanced, robust, and computationally efficient framework for evaluating modern LLMs in the creative era, with potential real-time applications in decoding and hallucination mitigation.

Abstract

Reference-free metrics like self-perplexity are strongly biased against creative text generation. We propose the Confidence Score (CS), derived from a model's output probability distribution, as a less biased alternative. Experiments on gpt-4o-mini show that while fluency-based metrics prefer novel responses in 0\% of cases on 99 creative prompts, our CS does so 19% of the time, a statistically significant difference (95% CI for difference: [11.1%, 27.3%]). We also show that CS effectively distinguishes between easy, medium, and hard tasks, confirmed by non-overlapping confidence intervals. The Confidence Score thus mitigates the creativity bias of traditional metrics while retaining their core evaluative strengths, offering a more balanced assessment for modern LLMs.