DIRAS: Efficient LLM Annotation of Document Relevance in Retrieval Augmented Generation

TL;DR

DIRAS presents a scalable framework to fine-tune open-source LLMs into calibrated domain-specific IR annotators, targeting recall in Retrieval Augmented Generation for integrative queries. The three-step pipeline—sampling query-document pairs, defining domain relevance, and distilling teacher judgments into student models—enables small models to produce binary relevance labels with calibrated scores. Through the ChatReportRetrieve dataset and real-world ClimRetrieve/QA experiments, DIRAS demonstrates that pointwise annotation with token-based calibration yields robust, GPT-4–level performance, reduces annotation bias, and supports IR benchmarking. The work argues for threshold-based retrieval and domain-aware relevance definitions to improve IR recall and practical deployment in RAG systems.

Abstract

Retrieval Augmented Generation (RAG) is widely employed to ground responses to queries on domain-specific documents. But do RAG implementations leave out important information when answering queries that need an integrated analysis of information (e.g., Tell me good news in the stock market today.)? To address these concerns, RAG developers need to annotate information retrieval (IR) data for their domain of interest, which is challenging because (1) domain-specific queries usually need nuanced definitions of relevance beyond shallow semantic relevance; and (2) human or GPT-4 annotation is costly and cannot cover all (query, document) pairs (i.e., annotation selection bias), thus harming the effectiveness in evaluating IR recall. To address these challenges, we propose DIRAS (Domain-specific Information Retrieval Annotation with Scalability), a manual-annotation-free schema that fine-tunes open-sourced LLMs to consider nuanced relevance definition and annotate (partial) relevance labels with calibrated relevance scores. Extensive evaluation shows that DIRAS enables smaller (8B) LLMs to achieve GPT-4-level performance on annotating and ranking unseen (query, document) pairs, and is helpful for real-world RAG development. All code, LLM generations, and human annotations can be found in \url{https://github.com/EdisonNi-hku/DIRAS}.

Figures (16)

• Figure 1: Overview of the functionality of DIRAS taking (query, relevance definition, document) triplets as input and output a binary verdict and a well-calibrated relevance score, which is sensitive to the grey-scale of partial relevance.
• Figure 2: DIRAS pipeline. Domain-specific queries, and documents as inputs; calibrated student LLM annotators as outputs.
• Figure 3: Our prompt template $\mathcal{P}$ for distilling training data from $\mathcal{M}_{t}$. "[Reason]" is only used in the CoT setup. It is shortened for presentation. Full $\mathcal{P}$ is in App. \ref{['fig:diras_prompt']}.
• Figure 4: Shaded bars denote the performance of original models. Colored bars denote the improvement brought by fine-tuning.
• Figure 5: The proximate amount of relevant information for 16 questions in all ClimRetrieve reports, according to Llama3-Tok's relevance scores.