Empowering Large Language Models to Set up a Knowledge Retrieval Indexer via Self-Learning
Xun Liang, Simin Niu, Zhiyu li, Sensen Zhang, Shichao Song, Hanyu Wang, Jiawei Yang, Feiyu Xiong, Bo Tang, Chenyang Xi
TL;DR
This work tackles the cost and noise challenges of building knowledge repositories for retrieval-augmented generation by introducing PG-RAG, a pre-retrieval framework in which LLMs autonomously read raw material and construct a hierarchical pseudo-graph of topics, routes, and facts. The method formalizes a graph $\mathcal{PG}$ with mind maps $\mathcal{M}$ and cross-document links, and defines a retrieval objective to produce a sub-graph $\mathcal{Z} \subseteq \mathcal{PG}$ that supports generation via $p(\mathcal{Z}|q) \cdot p(o|q, \mathcal{Z})$. Key contributions include (i) a generalizable, knowledge-driven indexing paradigm that reduces preset constraints; (ii) cross-document knowledge construction by clustering mind maps into super-nodes and learning fact-path embeddings; and (iii) a structured evidence mechanism that transforms queries into Key Points (KPs) and uses a matrix-based recall pipeline (template, pathway, control, staining, and aggregate matrices) to localize evidence efficiently, with $Z$ determined by $Z = \text{topK}(\sum_v CM^{v^{seed}} \odot PM^{v^{seed}})$. Experimental results on the CRUD-RAG benchmark show substantial improvements in single- and multi-document QA metrics (e.g., BLEU and QE-F1), demonstrating stronger, more scalable knowledge grounding for LLMs in knowledge-intensive tasks. The approach offers practical impact for deploying accurate, fast RAG systems across domains where cross-document reasoning is essential.
Abstract
Retrieval-Augmented Generation (RAG) offers a cost-effective approach to injecting real-time knowledge into large language models (LLMs). Nevertheless, constructing and validating high-quality knowledge repositories require considerable effort. We propose a pre-retrieval framework named Pseudo-Graph Retrieval-Augmented Generation (PG-RAG), which conceptualizes LLMs as students by providing them with abundant raw reading materials and encouraging them to engage in autonomous reading to record factual information in their own words. The resulting concise, well-organized mental indices are interconnected through common topics or complementary facts to form a pseudo-graph database. During the retrieval phase, PG-RAG mimics the human behavior in flipping through notes, identifying fact paths and subsequently exploring the related contexts. Adhering to the principle of the path taken by many is the best, it integrates highly corroborated fact paths to provide a structured and refined sub-graph assisting LLMs. We validated PG-RAG on three specialized question-answering datasets. In single-document tasks, PG-RAG significantly outperformed the current best baseline, KGP-LLaMA, across all key evaluation metrics, with an average overall performance improvement of 11.6%. Specifically, its BLEU score increased by approximately 14.3%, and the QE-F1 metric improved by 23.7%. In multi-document scenarios, the average metrics of PG-RAG were at least 2.35% higher than the best baseline. Notably, the BLEU score and QE-F1 metric showed stable improvements of around 7.55% and 12.75%, respectively. Our code: https://github.com/IAAR-Shanghai/PGRAG.