HybridRAG: A Practical LLM-based ChatBot Framework based on Pre-Generated Q&A over Raw Unstructured Documents

TL;DR

HybridRAG presents a practical Retrieval-Augmented Generation framework that pre-generates a large QA bank from raw, unstructured PDFs using OCR and layout analysis, organizing content into a hierarchical chunking structure. At query time, user questions are first matched against the pre-generated QA bank via dense embeddings, enabling fast, direct answers when there is a close match; otherwise, the system aggregates relevant chunks and queries an LLM to generate a response. Evaluations on OHRBench show that HybridRAG achieves lower latency and higher answer quality than a standard RAG baseline, with notable gains in domains rich in non-textual content. The approach offers a scalable solution for real-world chatbots that must process large volumes of unstructured documents under limited compute resources.

Abstract

Retrieval-Augmented Generation (RAG) has emerged as a powerful approach for grounding Large Language Model (LLM)-based chatbot responses on external knowledge. However, existing RAG studies typically assume well-structured textual sources (e.g. Wikipedia or curated datasets) and perform retrieval and generation at query time, which can limit their applicability in real-world chatbot scenarios. In this paper, we present HybridRAG, a novel and practical RAG framework towards more accurate and faster chatbot responses. First, HybridRAG ingests raw, unstructured PDF documents containing complex layouts (text, tables, figures) via Optical Character Recognition (OCR) and layout analysis, and convert them into hierarchical text chunks. Then, it pre-generates a plausible question-answer (QA) knowledge base from the organized chunks using an LLM. At query time, user questions are matched against this QA bank to retrieve immediate answers when possible, and only if no suitable QA match is found does our framework fall back to an on-the-fly response generation. Experiments on OHRBench demonstrate that our HybridRAG provides higher answer quality and lower latency compared to a standard RAG baseline. We believe that HybridRAG could be a practical solution for real-world chatbot applications that must handle large volumes of unstructured documents and lots of users under limited computational resources.

Figures (3)

• Figure 1: The overview of our HybridRAG.
• Figure 2: Quality evaluation of the pre-generated QAs.
• Figure 3: Performance depending on different threshold.