HiQA: A Hierarchical Contextual Augmentation RAG for Multi-Documents QA

TL;DR

HiQA is presented, an advanced multi-document question-answering (MDQA) framework that integrates cascading metadata into content and a multi-route retrieval mechanism that demonstrates the state-of-the-art performance in multi-document environments.

Abstract

Retrieval-augmented generation (RAG) has rapidly advanced the language model field, particularly in question-answering (QA) systems. By integrating external documents during the response generation phase, RAG significantly enhances the accuracy and reliability of language models. This method elevates the quality of responses and reduces the frequency of hallucinations, where the model generates incorrect or misleading information. However, these methods exhibit limited retrieval accuracy when faced with numerous indistinguishable documents, presenting notable challenges in their practical application. In response to these emerging challenges, we present HiQA, an advanced multi-document question-answering (MDQA) framework that integrates cascading metadata into content and a multi-route retrieval mechanism. We also release a benchmark called MasQA to evaluate and research in MDQA. Finally, HiQA demonstrates the state-of-the-art performance in multi-document environments.

Figures (16)

• Figure 1: Illustration of the proposed contextual text enhancement. The contextual structure can improve text alignment with the query for better matching in multi-documents scenarios.
• Figure 2: Experimental validation of performance degradation in multi-documents QA scenario. Testing with 88 documents, each containing one of 88 questions. Using a vanilla RAG and GPT-4 setup (chunk size=400, top-k=5). Only one incorrect answer when querying each question on a single document. However, querying all 88 documents together leads to 30 incorrect answers, demonstrating significant degradation as the number of documents increases.
• Figure 3: HiQA Framework. Illustration of the proposed framework. Initially, each document undergoes processing by a Markdown Formatter, transforming it into [chapter metadata: chapter content] pairs (termed segments) according to its inherent chapter structure, and is then stored in Markdown format. Subsequently, we extract the segment's hierarchy, and metadata is cascaded into each chapter, to build our database. Finally, we apply a Multi-Route retrieval method to enhance the RAG. Since hierarchical augmentation precedes retrieval, it offers a scalable solution to integrate with various embedding or retrieval methodologies seamlessly.
• Figure 4: Markdown Formatter. This demonstrates the extraction of chapter metadata and associated content from a long document and ensures alignment under sliding window processing.
• Figure 5: An Example Query-Answering on Texas Instruments Dataset