Chatbot Arena Meets Nuggets: Towards Explanations and Diagnostics in the Evaluation of LLM Responses

TL;DR

This work reframes the evaluation of retrieval-augmented LLMs by applying an automated nugget-based methodology to Search Arena battles. By extracting atomic nuggets from queries, retrieved content, and model responses, the Auto-Nuggetizer provides interpretable, diagnostic signals that align with human preferences, especially for knowledge-intensive queries. The study demonstrates that nugget score differences correlate with winner/loser/ tie outcomes, explores inversions across query types and languages, and shows that nugget generation from LLM outputs alone remains viable when URLs are unavailable. It also compares nugget-based evaluation to an LLM-as-a-judge approach, finding the latter less reliable for detecting ties and more prone to inversions, underscoring nugget-based evaluation as a promising, explainable tool for diagnosing and improving RAG systems.

Abstract

Battles, or side-by-side comparisons in so-called arenas that elicit human preferences, have emerged as a popular approach for assessing the output quality of LLMs. Recently, this idea has been extended to retrieval-augmented generation (RAG) systems. While undoubtedly representing an advance in evaluation, battles have at least two drawbacks, particularly in the context of complex information-seeking queries: they are neither explanatory nor diagnostic. Recently, the nugget evaluation methodology has emerged as a promising approach to evaluate the quality of RAG answers. Nuggets decompose long-form LLM-generated answers into atomic facts, highlighting important pieces of information necessary in a "good" response. In this work, we apply our AutoNuggetizer framework to analyze data from roughly 7K Search Arena battles provided by LMArena in a fully automatic manner. Our results show a significant correlation between nugget scores and human preferences, showcasing promise in our approach to explainable and diagnostic system evaluations. All the code necessary to reproduce results in our work is available in https://github.com/castorini/lmsys_nuggetize.

Figures (11)

• Figure 1: An end-to-end example from Search Arena illustrating both nugget generation and assignment. First, GPT4.1 generates nuggets based on the query, retrieved chunks from URL contents, and the responses from both models. Each nugget is then labeled with an importance level—either "vital" or "okay". Next, GPT4.1 evaluates whether each model supports each nugget, assigning one of three labels: "support", "partial support", or "no support". Finally, these support judgments are scored and aggregated to determine the overall outcome (the model with the higher score is preferred).
• Figure 2: Empirical probability density function (PDF) of nugget score differences ($\textrm{score}_B$$-$$\textrm{score}_A$) grouped by human preference category: model$_A$ wins, tie, or model$_B$ wins. A Kernel Density Estimation (KDE) with a bandwidth of 0.5 is fitted separately for each group.
• Figure 3: Empirical cumulative distribution functions (CDFs) comparing nugget score differences ($\textrm{score}_B$$-$$\textrm{score}_A$) across human vote categories. Each subplot shows a Kolmogorov-Smirnov (K-S) test between two groups: (left) model$_A$ wins vs. model$_B$ wins, (center) model$_A$ wins vs. tie, and (right) model$_B$ wins vs. tie. The K-S statistic and corresponding $p$-value are annotated in each plot, quantifying the distributional differences between groups.
• Figure 4: Confusion matrix comparing human and nugget preferences. A threshold of 0.07 is applied to treat nugget preference scores as a tie.
• Figure 5: Confusion matrices comparing human and nugget preferences across eight query classes from the Search Arena dataset. A threshold of 0.07 is used to treat nugget preference scores as a tie.