Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Say Less, Mean More: Leveraging Pragmatics in Retrieval-Augmented Generation

Haris Riaz, Ellen Riloff, Mihai Surdeanu

TL;DR

The paper addresses information overload in retrieval augmented generation by injecting pragmatics into the retrieval process. It introduces an unsupervised, pre retrieval method that identifies and highlights the most relevant sentences within retrieved documents, guided by Gricean maxims and using Step-Back prompting to drive iterative evidence collection. The approach is plug and play withDense Passage Retrieval and is evaluated across ARC-Challenge, PubHealth, and PopQA with five LLMs, showing notable gains up to nearly 20% in certain settings, while also outlining limitations and trade offs. The work potentially elevates RAG utility in open domain QA by focusing attention on high value evidence without discarding the full retrieved context, enabling more reliable and explainable downstream reasoning.

Abstract

We propose a simple, unsupervised method that injects pragmatic principles in retrieval-augmented generation (RAG) frameworks such as Dense Passage Retrieval to enhance the utility of retrieved contexts. Our approach first identifies which sentences in a pool of documents retrieved by RAG are most relevant to the question at hand, cover all the topics addressed in the input question and no more, and then highlights these sentences within their context, before they are provided to the LLM, without truncating or altering the context in any other way. We show that this simple idea brings consistent improvements in experiments on three question answering tasks (ARC-Challenge, PubHealth and PopQA) using five different LLMs. It notably enhances relative accuracy by up to 19.7% on PubHealth and 10% on ARC-Challenge compared to a conventional RAG system.

Say Less, Mean More: Leveraging Pragmatics in Retrieval-Augmented Generation

TL;DR

The paper addresses information overload in retrieval augmented generation by injecting pragmatics into the retrieval process. It introduces an unsupervised, pre retrieval method that identifies and highlights the most relevant sentences within retrieved documents, guided by Gricean maxims and using Step-Back prompting to drive iterative evidence collection. The approach is plug and play withDense Passage Retrieval and is evaluated across ARC-Challenge, PubHealth, and PopQA with five LLMs, showing notable gains up to nearly 20% in certain settings, while also outlining limitations and trade offs. The work potentially elevates RAG utility in open domain QA by focusing attention on high value evidence without discarding the full retrieved context, enabling more reliable and explainable downstream reasoning.

Abstract

We propose a simple, unsupervised method that injects pragmatic principles in retrieval-augmented generation (RAG) frameworks such as Dense Passage Retrieval to enhance the utility of retrieved contexts. Our approach first identifies which sentences in a pool of documents retrieved by RAG are most relevant to the question at hand, cover all the topics addressed in the input question and no more, and then highlights these sentences within their context, before they are provided to the LLM, without truncating or altering the context in any other way. We show that this simple idea brings consistent improvements in experiments on three question answering tasks (ARC-Challenge, PubHealth and PopQA) using five different LLMs. It notably enhances relative accuracy by up to 19.7% on PubHealth and 10% on ARC-Challenge compared to a conventional RAG system.

Paper Structure

This paper contains 16 sections, 3 equations, 2 figures, 8 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Approach: Combining Step-Back Reasoning With Pragmatic Retrieval
  4. Step-Back Query Expansion
  5. Parallel Iterative Evidence Retrieval
  6. Results
  7. Analysis
  8. Conclusions
  9. Human Evaluation of Evidence Quality
  10. Evaluation Criteria
  11. Low Quality Evidence
  12. Step-Back Reasoning Examples
  13. Step-back Prompt for Query Expansion
  14. Step-back Prompt for MCQ Answer Choices
  15. Experimental Details
  16. ...and 1 more sections

Figures (2)

  • Figure 1: Our proposed method. Each query is concatenated with a more abstract Step-back version of itself synthesized by a Step-back LLM. This new query is used initiate multi-hop retrieval where in each hop the query is aligned with passages retrieved by DPR to select one evidence sentence. These sentences are aggregated across hops with alignment at each hop driven by query reformulation based on missing information (maxim of relation) between the current set of selected evidence sentences and current query. After all query keywords are covered by the retrieved evidences (maxim of quantity), our method highlights them within their original contexts and provides them to the LLM.
  • Figure 2: Performance of Qwen2.5-3B on: (top) ARC-Challenge, (middle) PopQA, (bottom) PubHealth under different Evidence Highlighting settings, with varying top-$k$ where $k$ is the number of DPR contexts retrieved.