Enhancing Retrieval in QA Systems with Derived Feature Association

TL;DR

The paper tackles the problem that retrieval in retrieval-augmented generation (RAG) often fails to surface implicit information needed for complex QA. It introduces RAIDD, a two-phase framework that derives features from ingested documents (summaries and questions) to guide retrieval, while preserving access to the original text. Four flavors (RAIDD-S, RAIDD-S ICL, RAIDD-Q, RAIDD-U) demonstrate how derived documents can enhance context selection and answer accuracy on long-context QA tasks, with improvements up to 15% in QA accuracy on the LooGLE dataset. The work highlights that enhanced retrieval must be matched by capable LLM generation, and points to future directions including richer derived-document types and further tuning of the derivation process to support cross-domain applicability.

Abstract

Retrieval augmented generation (RAG) has become the standard in long context question answering (QA) systems. However, typical implementations of RAG rely on a rather naive retrieval mechanism, in which texts whose embeddings are most similar to that of the query are deemed most relevant. This has consequences in subjective QA tasks, where the most relevant text may not directly contain the answer. In this work, we propose a novel extension to RAG systems, which we call Retrieval from AI Derived Documents (RAIDD). RAIDD leverages the full power of the LLM in the retrieval process by deriving inferred features, such as summaries and example questions, from the documents at ingest. We demonstrate that this approach significantly improves the performance of RAG systems on long-context QA tasks.

Figures (3)

• Figure 1: Example of a question from the LooGLE loogle dataset answered by a traditional RAG system and RAIDD-S. The traditional system identifies the a text that describes how Picardo's work was regarded by one figure, but it fails to identify the more subtly worded target text which contains the answer. Using summary generation, RAIDD-S is able to retrieve a more relevant passage and correctly answer the question.
• Figure 2: During the document ingest phase, RAIDD derives new documents from the input by prompting a GPT feature extractor to summarize and generate questions from the original documents. At inference, the retriever identifies the most relevant derived documents and places the corresponding source documents into context for question answering.
• Figure 3: In-context learning using an LLM as an optimizer