Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

A Hybrid RAG System with Comprehensive Enhancement on Complex Reasoning

Ye Yuan, Chengwu Liu, Jingyang Yuan, Gongbo Sun, Siqi Li, Ming Zhang

TL;DR

The paper presents a hybrid Retrieval-Augmented Generation system augmented with a comprehensive optimization suite to address domain knowledge gaps, hallucinations, and time-sensitive information. It integrates modules for web-page processing, attribute prediction, numerical calculation, LLM-based knowledge extraction, a knowledge graph interface, rigorous reasoning, and corner-case handling, all evaluated on the CRAG benchmark with strong local results and competitive private performance. The approach emphasizes modular improvements across retrieval, reasoning, and generation, showing significant gains in accuracy and reduced hallucinations, while acknowledging limitations in dynamic data handling and KG utilization. The work demonstrates practical potential for robust, up-to-date, and explainable QA systems in real-world settings and provides code availability for reproducibility.

Abstract

Retrieval-augmented generation (RAG) is a framework enabling large language models (LLMs) to enhance their accuracy and reduce hallucinations by integrating external knowledge bases. In this paper, we introduce a hybrid RAG system enhanced through a comprehensive suite of optimizations that significantly improve retrieval quality, augment reasoning capabilities, and refine numerical computation ability. We refined the text chunks and tables in web pages, added attribute predictors to reduce hallucinations, conducted LLM Knowledge Extractor and Knowledge Graph Extractor, and finally built a reasoning strategy with all the references. We evaluated our system on the CRAG dataset through the Meta CRAG KDD Cup 2024 Competition. Both the local and online evaluations demonstrate that our system significantly enhances complex reasoning capabilities. In local evaluations, we have significantly improved accuracy and reduced error rates compared to the baseline model, achieving a notable increase in scores. In the meanwhile, we have attained outstanding results in online assessments, demonstrating the performance and generalization capabilities of the proposed system. The source code for our system is released in \url{https://gitlab.aicrowd.com/shizueyy/crag-new}.

A Hybrid RAG System with Comprehensive Enhancement on Complex Reasoning

TL;DR

The paper presents a hybrid Retrieval-Augmented Generation system augmented with a comprehensive optimization suite to address domain knowledge gaps, hallucinations, and time-sensitive information. It integrates modules for web-page processing, attribute prediction, numerical calculation, LLM-based knowledge extraction, a knowledge graph interface, rigorous reasoning, and corner-case handling, all evaluated on the CRAG benchmark with strong local results and competitive private performance. The approach emphasizes modular improvements across retrieval, reasoning, and generation, showing significant gains in accuracy and reduced hallucinations, while acknowledging limitations in dynamic data handling and KG utilization. The work demonstrates practical potential for robust, up-to-date, and explainable QA systems in real-world settings and provides code availability for reproducibility.

Abstract

Retrieval-augmented generation (RAG) is a framework enabling large language models (LLMs) to enhance their accuracy and reduce hallucinations by integrating external knowledge bases. In this paper, we introduce a hybrid RAG system enhanced through a comprehensive suite of optimizations that significantly improve retrieval quality, augment reasoning capabilities, and refine numerical computation ability. We refined the text chunks and tables in web pages, added attribute predictors to reduce hallucinations, conducted LLM Knowledge Extractor and Knowledge Graph Extractor, and finally built a reasoning strategy with all the references. We evaluated our system on the CRAG dataset through the Meta CRAG KDD Cup 2024 Competition. Both the local and online evaluations demonstrate that our system significantly enhances complex reasoning capabilities. In local evaluations, we have significantly improved accuracy and reduced error rates compared to the baseline model, achieving a notable increase in scores. In the meanwhile, we have attained outstanding results in online assessments, demonstrating the performance and generalization capabilities of the proposed system. The source code for our system is released in \url{https://gitlab.aicrowd.com/shizueyy/crag-new}.
Paper Structure (30 sections, 1 equation, 4 figures, 5 tables)

This paper contains 30 sections, 1 equation, 4 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. Pre-retrieval
  4. Retrieval
  5. Post-retrieval
  6. Generation
  7. CRAG Benchmark
  8. System Design
  9. Web Page Processing
  10. Attribute Predictor
  11. Numerical Calculator
  12. LLM Knowledge Extractor
  13. Knowledge Graph Module
  14. Reasoning Module
  15. Handling Corner Cases
  16. ...and 15 more sections

Figures (4)

  • Figure 1: The complete design of our system. There are two possible routes for the generation. If the query is classified by the in-context learning as "dynamic", we will output "I don't know" directly to reduce hallucination on these hard problems.
  • Figure 2: The design of our web page processing. We utilized Trafilatura and BeautifulSoup to extract plain text and tables from web pages. Following this extraction, we employed Blingfire to segment the plain text into sentences, which were then grouped into chunks based on heuristic methods. Additionally, the tables were converted into Markdown format for further processing.
  • Figure 3: Detailed score across different attributes in the local evaluation of Task 1.
  • Figure 4: Detailed score across different attributes in the online private evaluation Task 1.