Insight-RAG: Enhancing LLMs with Insight-Driven Augmentation

TL;DR

Insight-RAG addresses core limitations of traditional retrieval-augmented generation by inserting an insight extraction stage before retrieval. The framework comprises three units—Insight Identifier, Insight Miner (CPT-LoRA on the target-domain corpus using Llama-3.2 3B), and a final Response Generator—to produce contextually enriched responses from identified insights. Evaluated on two scientific paper datasets (AAN and OC) with benchmarks for deeply buried, multi-source, and non-QA tasks, Insight-RAG achieves substantial gains, up to 60 percentage points in accuracy for certain tasks and up to 5.4 percentage points in non-QA tasks, outperforming conventional RAG across configurations. The work highlights how targeted insight retrieval expands RAG's applicability, offers detailed component analyses, and points to future directions including domain extension, hierarchical insight extraction, and multimodal integration.

Abstract

Retrieval Augmented Generation (RAG) frameworks have shown significant promise in leveraging external knowledge to enhance the performance of large language models (LLMs). However, conventional RAG methods often retrieve documents based solely on surface-level relevance, leading to many issues: they may overlook deeply buried information within individual documents, miss relevant insights spanning multiple sources, and are not well-suited for tasks beyond traditional question answering. In this paper, we propose Insight-RAG, a novel framework designed to address these issues. In the initial stage of Insight-RAG, instead of using traditional retrieval methods, we employ an LLM to analyze the input query and task, extracting the underlying informational requirements. In the subsequent stage, a specialized LLM -- trained on the document database -- is queried to mine content that directly addresses these identified insights. Finally, by integrating the original query with the retrieved insights, similar to conventional RAG approaches, we employ a final LLM to generate a contextually enriched and accurate response. Using two scientific paper datasets, we created evaluation benchmarks targeting each of the mentioned issues and assessed Insight-RAG against traditional RAG pipeline. Our results demonstrate that the Insight-RAG pipeline successfully addresses these challenges, outperforming existing methods by a significant margin in most cases. These findings suggest that integrating insight-driven retrieval within the RAG framework not only enhances performance but also broadens the applicability of RAG to tasks beyond conventional question answering.

Figures (8)

• Figure 1: In conventional RAG, using a retriever model, we first retrieve relevant documents to answer a question. In contrast, in Insight-RAG, we first identify necessary insights to solve the task (e.g., answering a question), and then feed the identified insights to an LLM continually pre-trained over the documents to extract the necessary insights before feeding them to the final LLM to solve the task.
• Figure 2: We create our benchmark in several steps: 1) extracting triples from domain-specific documents using GPT-4o mini and then manually normalizing/filtering them, 2) filtering the triples for each different type of issue, 3) using GPT-4o mini to translate the sampled triples to question format, asking about the object of the triple.
• Figure 3: The performance comparison of RAG versus Insight-RAG across the AAN and OC datasets in answering question based on deeply buried information. As demonstrated, DeepSeek-R1 performed the best, followed by Llama-3.3 70B. Moreover, we observe that Insight-RAG, even with only one generated insight, outperforms RAG-based solutions by a considerable margin. Additionally, while retrieving more documents reduces this performance gap, Insight-RAG maintains a significant advantage.
• Figure 4: The performance comparison of RAG versus Insight-RAG across the AAN and OC datasets in answering questions requiring information from multiple sources. As demonstrated, DeepSeek-R1 performed the best, followed by Llama-3.3 70B. Moreover, we observe that Insight-RAG with only a few generated insights achieves a much higher performance, with the performance continuing to improve at a reduced rate as more insights are added.
• Figure 5: Insight Identifier performance: We ask GPT-4o mini to score the identified insights compared to the gold insights using a three-point scale: 0 (not similar), 0.5 (partially similar), and 1 (completely similar).