Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

LANCER: LLM Reranking for Nugget Coverage

Jia-Huei Ju, François G. Landry, Eugene Yang, Suzan Verberne, Andrew Yates

TL;DR

LANCER tackles nugget coverage in long-form RAG by moving beyond pure relevance to optimize for information coverage using a three-stage LLM-based reranker. It generates synthetic sub-questions, assesses whether documents answer them via answerability judgments, and applies coverage-based aggregation to produce a reranked document set that maximizes Nugget Coverage. Across two nugget-annotated datasets, LANCER improves $\alpha$-nDCG and $Cov$ relative to baselines, with oracle sub-questions yielding substantial further gains and highlighting sub-question quality as crucial. The approach offers transparency through explicit sub-questions and judgments and provides practical guidance on aggregation strategies and parameter settings for long-form report generation.

Abstract

Unlike short-form retrieval-augmented generation (RAG), such as factoid question answering, long-form RAG requires retrieval to provide documents covering a wide range of relevant information. Automated report generation exemplifies this setting: it requires not only relevant information but also a more elaborate response with comprehensive information. Yet, existing retrieval methods are primarily optimized for relevance ranking rather than information coverage. To address this limitation, we propose LANCER, an LLM-based reranking method for nugget coverage. LANCER predicts what sub-questions should be answered to satisfy an information need, predicts which documents answer these sub-questions, and reranks documents in order to provide a ranked list covering as many information nuggets as possible. Our empirical results show that LANCER enhances the quality of retrieval as measured by nugget coverage metrics, achieving higher $α$-nDCG and information coverage than other LLM-based reranking methods. Our oracle analysis further reveals that sub-question generation plays an essential role.

LANCER: LLM Reranking for Nugget Coverage

TL;DR

LANCER tackles nugget coverage in long-form RAG by moving beyond pure relevance to optimize for information coverage using a three-stage LLM-based reranker. It generates synthetic sub-questions, assesses whether documents answer them via answerability judgments, and applies coverage-based aggregation to produce a reranked document set that maximizes Nugget Coverage. Across two nugget-annotated datasets, LANCER improves -nDCG and relative to baselines, with oracle sub-questions yielding substantial further gains and highlighting sub-question quality as crucial. The approach offers transparency through explicit sub-questions and judgments and provides practical guidance on aggregation strategies and parameter settings for long-form report generation.

Abstract

Unlike short-form retrieval-augmented generation (RAG), such as factoid question answering, long-form RAG requires retrieval to provide documents covering a wide range of relevant information. Automated report generation exemplifies this setting: it requires not only relevant information but also a more elaborate response with comprehensive information. Yet, existing retrieval methods are primarily optimized for relevance ranking rather than information coverage. To address this limitation, we propose LANCER, an LLM-based reranking method for nugget coverage. LANCER predicts what sub-questions should be answered to satisfy an information need, predicts which documents answer these sub-questions, and reranks documents in order to provide a ranked list covering as many information nuggets as possible. Our empirical results show that LANCER enhances the quality of retrieval as measured by nugget coverage metrics, achieving higher -nDCG and information coverage than other LLM-based reranking methods. Our oracle analysis further reveals that sub-question generation plays an essential role.
Paper Structure (26 sections, 4 equations, 6 figures, 2 tables)

This paper contains 26 sections, 4 equations, 6 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Preliminaries
  4. Method: LANCER
  5. Synthetic Sub-question Generation.
  6. Answerability Judgments Generation.
  7. Coverage-based Aggregation Strategies.
  8. Summation (sum & sum-$\tau$).
  9. Reciprocal Rank Fusion (RRF).
  10. Greedy Utility Selection (greedy-sum, greedy-$\alpha$, & greedy-$cov$).
  11. Experiments and Results
  12. Experimental Setup
  13. Evaluation Datasets.
  14. Evaluation Methods.
  15. First-stage Retrieval.
  16. ...and 11 more sections

Figures (6)

  • Figure 1: LANCER consists of three stages (blue boxes). The final retrieved context $Z$ is evaluated with nugget coverage metrics.
  • Figure 2: Sub-question generation prompt to produce a list of sub-questions.
  • Figure 3: Rubric-based answerability judgment prompt. The output rating is converted into 0 to 5, and the output with incorrect formats is assigned to 0.
  • Figure 4: Coverage ($Cov$) grows with respect to the top-$k$ cutoff on NeuCLIR'24 ReportGen evaluation data. Each line indicates the retrieved contexts from different retrieval pipelines.
  • Figure 5: Evaluation results on the NeuCLIR’24 ReportGen with different numbers of synthetic sub-questions ($x$-axis). We use the sum strategy for all the settings. The colors indicate the three first-stage retrieval.
  • ...and 1 more figures