Initial Nugget Evaluation Results for the TREC 2024 RAG Track with the AutoNuggetizer Framework

TL;DR

An initial look at partial results from the TREC 2024 Retrieval-Augmented Generation (RAG) Track is provided, and a strong correlation between scores derived from a fully automatic nugget evaluation and a (mostly) manual nugget evaluation by human assessors is observed.

Abstract

This report provides an initial look at partial results from the TREC 2024 Retrieval-Augmented Generation (RAG) Track. We have identified RAG evaluation as a barrier to continued progress in information access (and more broadly, natural language processing and artificial intelligence), and it is our hope that we can contribute to tackling the many challenges in this space. The central hypothesis we explore in this work is that the nugget evaluation methodology, originally developed for the TREC Question Answering Track in 2003, provides a solid foundation for evaluating RAG systems. As such, our efforts have focused on "refactoring" this methodology, specifically applying large language models to both automatically create nuggets and to automatically assign nuggets to system answers. We call this the AutoNuggetizer framework. Within the TREC setup, we are able to calibrate our fully automatic process against a manual process whereby nuggets are created by human assessors semi-manually and then assigned manually to system answers. Based on initial results across 21 topics from 45 runs, we observe a strong correlation between scores derived from a fully automatic nugget evaluation and a (mostly) manual nugget evaluation by human assessors. This suggests that our fully automatic evaluation process can be used to guide future iterations of RAG systems.

Figures (7)

• Figure 1: Examples of five topics taken from the TREC 2024 RAG Track, demonstrating the real-world user queries.
• Figure 2: The high-level overview of the TREC 2024 RAG Track.
• Figure 3: Prompt for the iterative LLM-based nuggetization at turn $i$.
• Figure 4: Prompt for determining the importance of nuggets. At each turn, at most 10 nuggets are passed to the LLM.
• Figure 5: Prompt for nugget assignment. At each turn, at most 10 nuggets are passed to the LLM.