Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

URAG: Implementing a Unified Hybrid RAG for Precise Answers in University Admission Chatbots -- A Case Study at HCMUT

Long Nguyen, Tho Quan

TL;DR

URAG tackles hallucinations in university admissions chatbots by fusing a rule-based FAQ tier with a retrieved-document tier in a two-tier hybrid RAG. It introduces URAG-D for document augmentation and URAG-F for FAQ enrichment, leverages Chain-of-Thought prompting, and uses a lightweight Vietnamese LLM for generation. In experiments with 500 questions and 300 documents, URAG achieves competitive accuracy against SOTA commercial models and a four-month deployment at HCMUT demonstrates practical viability with fast, traceable responses. Limitations include reliance on a small generator and opportunities for future work in Hybrid Search and domain-specific fine-tuning, with human oversight for edge cases.

Abstract

With the rapid advancement of Artificial Intelligence, particularly in Natural Language Processing, Large Language Models (LLMs) have become pivotal in educational question-answering systems, especially university admission chatbots. Concepts such as Retrieval-Augmented Generation (RAG) and other advanced techniques have been developed to enhance these systems by integrating specific university data, enabling LLMs to provide informed responses on admissions and academic counseling. However, these enhanced RAG techniques often involve high operational costs and require the training of complex, specialized modules, which poses challenges for practical deployment. Additionally, in the educational context, it is crucial to provide accurate answers to prevent misinformation, a task that LLM-based systems find challenging without appropriate strategies and methods. In this paper, we introduce the Unified RAG (URAG) Framework, a hybrid approach that significantly improves the accuracy of responses, particularly for critical queries. Experimental results demonstrate that URAG enhances our in-house, lightweight model to perform comparably to state-of-the-art commercial models. Moreover, to validate its practical applicability, we conducted a case study at our educational institution, which received positive feedback and acclaim. This study not only proves the effectiveness of URAG but also highlights its feasibility for real-world implementation in educational settings.

URAG: Implementing a Unified Hybrid RAG for Precise Answers in University Admission Chatbots -- A Case Study at HCMUT

TL;DR

URAG tackles hallucinations in university admissions chatbots by fusing a rule-based FAQ tier with a retrieved-document tier in a two-tier hybrid RAG. It introduces URAG-D for document augmentation and URAG-F for FAQ enrichment, leverages Chain-of-Thought prompting, and uses a lightweight Vietnamese LLM for generation. In experiments with 500 questions and 300 documents, URAG achieves competitive accuracy against SOTA commercial models and a four-month deployment at HCMUT demonstrates practical viability with fast, traceable responses. Limitations include reliance on a small generator and opportunities for future work in Hybrid Search and domain-specific fine-tuning, with human oversight for edge cases.

Abstract

With the rapid advancement of Artificial Intelligence, particularly in Natural Language Processing, Large Language Models (LLMs) have become pivotal in educational question-answering systems, especially university admission chatbots. Concepts such as Retrieval-Augmented Generation (RAG) and other advanced techniques have been developed to enhance these systems by integrating specific university data, enabling LLMs to provide informed responses on admissions and academic counseling. However, these enhanced RAG techniques often involve high operational costs and require the training of complex, specialized modules, which poses challenges for practical deployment. Additionally, in the educational context, it is crucial to provide accurate answers to prevent misinformation, a task that LLM-based systems find challenging without appropriate strategies and methods. In this paper, we introduce the Unified RAG (URAG) Framework, a hybrid approach that significantly improves the accuracy of responses, particularly for critical queries. Experimental results demonstrate that URAG enhances our in-house, lightweight model to perform comparably to state-of-the-art commercial models. Moreover, to validate its practical applicability, we conducted a case study at our educational institution, which received positive feedback and acclaim. This study not only proves the effectiveness of URAG but also highlights its feasibility for real-world implementation in educational settings.

Paper Structure

This paper contains 17 sections, 2 equations, 6 figures, 4 tables, 3 algorithms.

Table of Contents

  1. Introduction
  2. Related Work
  3. Retrieval-Augmented Generation (RAG)
  4. Strategies for Enhancing a RAG Pipeline
  5. Practical Approaches to University Admission Chatbots
  6. URAG: A Unified RAG for Precise University Admission Chatbots
  7. Overview of URAG Architecture
  8. The Truth Behind "Unified" in URAG Architecture
  9. Experimentation
  10. Experiment Setup
  11. Methodology
  12. Results and Analysis
  13. Case Study: Deployment of the HCMUT Admission Chatbot
  14. Discussion, Limitations, and Future Work
  15. Appendix
  16. ...and 2 more sections

Figures (6)

  • Figure 1: The architecture of URAG framework illustrating the two-tiered approach for improving LLM performance in university admission chatbots.
  • Figure 2: An example of a typical RAG pipeline.
  • Figure 3: Illustration of the URAG framework, highlighting the two-tiered approach.
  • Figure 4: Illustrative overview of the two mechanisms implemented during the preparatory phase of URAG.
  • Figure 5: Comprehensive Analysis of Chatbot Model Performance and URAG's Tiers
  • ...and 1 more figures