SuRe: Summarizing Retrievals using Answer Candidates for Open-domain QA of LLMs

TL;DR

SuRe tackles open-domain QA with LLMs by generating multiple answer candidates and building candidate-conditioned summaries from retrieved passages. It then evaluates the summaries for validity and relative informativeness to select the final answer, all via zero-shot prompts. Experiments across four datasets and multiple retrievers/LLMs show consistent improvements in EM and F1, and the generated summaries also serve as meaningful rationales for model and human readers. The approach is retrieval-agnostic and prompts-only, enabling broad applicability to real-world systems using black-box LLMs.

Abstract

Large language models (LLMs) have made significant advancements in various natural language processing tasks, including question answering (QA) tasks. While incorporating new information with the retrieval of relevant passages is a promising way to improve QA with LLMs, the existing methods often require additional fine-tuning which becomes infeasible with recent LLMs. Augmenting retrieved passages via prompting has the potential to address this limitation, but this direction has been limitedly explored. To this end, we design a simple yet effective framework to enhance open-domain QA (ODQA) with LLMs, based on the summarized retrieval (SuRe). SuRe helps LLMs predict more accurate answers for a given question, which are well-supported by the summarized retrieval that could be viewed as an explicit rationale extracted from the retrieved passages. Specifically, SuRe first constructs summaries of the retrieved passages for each of the multiple answer candidates. Then, SuRe confirms the most plausible answer from the candidate set by evaluating the validity and ranking of the generated summaries. Experimental results on diverse ODQA benchmarks demonstrate the superiority of SuRe, with improvements of up to 4.6% in exact match (EM) and 4.0% in F1 score over standard prompting approaches. SuRe also can be integrated with a broad range of retrieval methods and LLMs. Finally, the generated summaries from SuRe show additional advantages to measure the importance of retrieved passages and serve as more preferred rationales by models and humans.

Figures (17)

• Figure 1: Zero-shot QA accuracy with various LLMs on Natural Question kwiatkowski2019natural. The performances of LLaMA-33B, GLaM-62B, and PaLM-540B are from the corresponding papers, respectively chowdhery2022palmdu2022glamtouvron2023llama1.
• Figure 2: Example of QA with the proposed SuRe framework. Given a query question and relevant passages retrieved by an external method, e.g., BM25 robertson2009probabilistic, a large language model, e.g., ChatGPT, needs to predict the answer. To improve this, SuRe first generates multiple answer candidates via prompting, and then conditionally summarizes the retrieved passages to support each candidate. By comparing the validity and relative informativeness of summaries, SuRe selects the most plausible candidate as a final prediction.
• Figure 3: TF-IDF overlap between candidates and conditional summarizations.
• Figure 4: (a) EM with different numbers of retrieved passages ($N$) under ChatGPT and BM25. (b) Comparison between SuRe's summarization and generic summarization via GPT-4 evaluation liu2023gpteval. (c) Human preference between SuRe's summarization and generic summarization on 84 samples of NQ$^{*}$, along with GPT-4 evaluation. More results are in Appendix \ref{['appendixE']}.
• Figure 5: Qualitative comparison of candidate-conditioned summarization from SuRe (Ours) compared to generic summarization as a rationale for the answer. More examples are in Appendix \ref{['appendixC']}.