Prompt-Guided Retrieval Augmentation for Non-Knowledge-Intensive Tasks

TL;DR

PGRA extends retrieval-augmented methods from mainly knowledge-intensive tasks to non-knowledge-intensive tasks by a two-stage design: a shared task-agnostic retriever builds a static external index, and a prompt-guided reranker tailors evidence relevance per task using in-context PLM representations, followed by a FiD reader. This approach enables diverse relevance score functions without retraining retrievers, and achieves strong results across SST, CoLA, TREC, and sentiment/subjectivity tasks while keeping training costs low. The paper provides extensive analyses on label-consistency, evidence granularity, and retriever generalization, demonstrating robustness and practical applicability. Overall, PGRA offers a scalable, low-cost framework for effective retrieval augmentation tailored to a broad range of NKI tasks, with clear avenues for further improvement and ethical considerations.

Abstract

Retrieval-augmented methods have received increasing attention to support downstream tasks by leveraging useful information from external resources. Recent studies mainly focus on exploring retrieval to solve knowledge-intensive (KI) tasks. However, the potential of retrieval for most non-knowledge-intensive (NKI) tasks remains under-explored. There are two main challenges to leveraging retrieval-augmented methods for NKI tasks: 1) the demand for diverse relevance score functions and 2) the dilemma between training cost and task performance. To address these challenges, we propose a two-stage framework for NKI tasks, named PGRA. In the first stage, we adopt a task-agnostic retriever to build a shared static index and select candidate evidence efficiently. In the second stage, we design a prompt-guided reranker to rerank the nearest evidence according to task-specific relevance for the reader. Experimental results show that PGRA outperforms other state-of-the-art retrieval-augmented methods. Our analyses further investigate the influence factors to model performance and demonstrate the generality of PGRA. Codes are available at https://github.com/THUNLP-MT/PGRA.

Figures (6)

• Figure 1: The framework of our proposed Prompt-Guided Retrieval Augmentation (PGRA) method. We first retrieve candidates through a task-agnostic retriever (Section \ref{['sec21']}), then use a task-specific prompt and pre-trained language model (PLM) to rerank the candidates (Section \ref{['sec22']}). We send the top results to the reader to make predictions. (Section \ref{['sec23']}).
• Figure 2: The pseudo label consistency of samples in SST-2 with PGRA and FiD (T5-base models for both). We plot the accuracy scores of instances with different numbers of label-consistent evidence, along with the number of such instances.
• Figure 3: Accuracy against $k$ (left) and $d$ (right). Details of performance on different tasks can be found in \ref{['app:kdsize']}.
• Figure 4: Average performance and average consistency score on 5 tasks (SST-2, SST-5, CoLA, MR and MPQA) against different OPT model sizes. Detailed information can be found in Appendix \ref{['app:kdsize']}.
• Figure 5: Performance of PGRA with passage-level and sentence-level external datastores.