Knowledge Pyramid Construction for Multi-Level Retrieval-Augmented Generation

TL;DR

The paper tackles the precision-recall trade-off in retrieval-augmented QA by introducing PolyRAG, a three-layer knowledge pyramid (Ontology, Knowledge Graph, and raw text) that enables cross-layer augmentation and condensation within a waterfall retrieval. It presents two domain benchmarks (AcadChall and an extended FiQA) to validate the approach, showing state-of-the-art performance and substantial gains for models like GPT-4. Key contributions include the construction and interaction of the Pyramid layers, novel knowledge completion and condensation mechanisms, and empirical evidence that higher layers (Ontology and KG) substantially improve precision and recall. The work advances domain-specific QA by enabling more compact yet comprehensive knowledge representations and efficient query processing, with practical impact on real-world, knowledge-intensive applications.

Abstract

This paper addresses the need for improved precision in existing knowledge-enhanced question-answering frameworks, specifically Retrieval-Augmented Generation (RAG) methods that primarily focus on enhancing recall. We propose a multi-layer knowledge pyramid approach within the RAG framework to achieve a better balance between precision and recall. The knowledge pyramid consists of three layers: Ontologies, Knowledge Graphs (KGs), and chunk-based raw text. We employ cross-layer augmentation techniques for comprehensive knowledge coverage and dynamic updates of the Ontology schema and instances. To ensure compactness, we utilize cross-layer filtering methods for knowledge condensation in KGs. Our approach, named PolyRAG, follows a waterfall model for retrieval, starting from the top of the pyramid and progressing down until a confident answer is obtained. We introduce two benchmarks for domain-specific knowledge retrieval, one in the academic domain and the other in the financial domain. The effectiveness of the methods has been validated through comprehensive experiments by outperforming 19 SOTA methods. An encouraging observation is that the proposed method has augmented the GPT-4, providing 395% F1 gain by improving its performance from 0.1636 to 0.8109.

Figures (6)

• Figure 1: PolyRAG Framework based on the concept of Knowledge Pyramid: different layers signify varying levels of specificity and richness. The layers are mutually complementary, working together for a balance between recall and precision.
• Figure 2: The framework of PolyRAG includes: (1) Knowledge Pyramid Construction, starting from extracting knowledge from corpora data to form the initial layers, then performing the layer interactions, which are knowledge completion and condensation. (2) Given the refined Knowledge Pyramid, the LLM inference obtains the contexts through a multi-level query, which retrieves the knowledge from the higher layer down to the lower layer in a waterfall pattern.
• Figure 3: Knowledge completion through the cross-layer interactions. The comprehensiveness of the initial Ontology schema has been improved with the noteworthy concepts identified from the knowledge graph layer.
• Figure 4: The distribution of KG and Ontology: the original distributions (left), samples with a high priority for knowledge completion are highlighted in red (middle), and the condensed KG after knowledge condensation (right).
• Figure 5: Illustration of multi-level querying: the two queries are addressed at the Ontology and Knowledge Graph layers, respectively, bypassing the lower levels.