Automated Evaluation of Retrieval-Augmented Language Models with Task-Specific Exam Generation

TL;DR

The paper addresses the challenge of evaluating Retrieval-Augmented LLMs on task-specific accuracy without ground-truth datasets by generating synthetic, task-aligned multiple-choice exams from the task corpus. It introduces a two-pronged methodology: automatic exam generation with rigorous quality filtering and a hierarchical Item Response Theory framework that decomposes model ability into LLM, retrieval, and in-context learning components, while weighting questions by informativeness. Experiments across four diverse domains (AWS DevOps, ArXiv abstracts, StackExchange, SEC filings) reveal that retrieval strategy often yields larger gains than merely increasing model size, and that the exam framework provides both predictive and prescriptive insights for RAG design. The work contributes a scalable, interpretable benchmarking framework with open-source tooling to standardize task-specific RAG evaluation and guide component co-design in real-world applications.

Abstract

We propose a new method to measure the task-specific accuracy of Retrieval-Augmented Large Language Models (RAG). Evaluation is performed by scoring the RAG on an automatically-generated synthetic exam composed of multiple choice questions based on the corpus of documents associated with the task. Our method is an automated, cost-efficient, interpretable, and robust strategy to select the optimal components for a RAG system. We leverage Item Response Theory (IRT) to estimate the quality of an exam and its informativeness on task-specific accuracy. IRT also provides a natural way to iteratively improve the exam by eliminating the exam questions that are not sufficiently informative about a model's ability. We demonstrate our approach on four new open-ended Question-Answering tasks based on Arxiv abstracts, StackExchange questions, AWS DevOps troubleshooting guides, and SEC filings. In addition, our experiments reveal more general insights into factors impacting RAG performance like size, retrieval mechanism, prompting and fine-tuning. Most notably, our findings show that choosing the right retrieval algorithms often leads to bigger performance gains than simply using a larger language model.

Figures (12)

• Figure 1: Granular results of our exam evaluation for the task of AWS DevOps troubleshooting. Accuracy is reported for different retrieval approaches and retriever sizes, on a % scale. Labels on the diameter shows the troubleshooting categories, i.e., AWS resources. Colors correspond to different retrieval approaches (Oracle, DPRV2, MultiQA, ClosedBook, as discussed in Section \ref{['sec:ragpipelines']}) and patterns correspond to the base LLM size (7B, 13B, and 70B). For instance, we observe that a small model such as Mistral-7B with MultiQA embeddings has an accuracy around 80% for the AWS resource "Relational Database Service" (RDS).
• Figure 2: Representation of Bloom's revised taxonomy. The cognitive complexity of skills increase from the bottom to the top of the pyramid. Source: bloom
• Figure 3: Aggregated Information function $\Bar{I}_{cat}(\theta)$ for $t_{\mathit{stk}}$, averaged according to Bloom taxonomy. Each cross on the x-axis correspond to a given model ability $\theta_{m}$, with no particular signification granted to their colors.
• Figure 4: Aggregated Information function $\Bar{I}_{cat}(\theta)$ for $t_{\mathit{stk}}$, averaged according to semantic taxonomy. Each cross on the x-axis correspond to a given model ability $\theta_{m}$.
• Figure 5: Evolution of Exam Informativeness during the iterative process, for ArXiv task. Each curve represents the exam aggregated Information function $\Bar{I}_{\mathcal{Q}_{i}}(\theta)$, at step $i$.