Leveraging the Domain Adaptation of Retrieval Augmented Generation Models for Question Answering and Reducing Hallucination

TL;DR

Investigation of the performance of diverse RAG and RAG-like architectures through domain adaptation and their ability to generate accurate and relevant response grounded in the contextual knowledge base indicates that domain adaptation not only enhances the models' performance on QA tasks but also significantly reduces hallucinations across all evaluated RAG architectures.

Abstract

While ongoing advancements in Large Language Models have demonstrated remarkable success across various NLP tasks, Retrieval Augmented Generation Model stands out to be highly effective on downstream applications like Question Answering. Recently, RAG-end2end model further optimized the architecture and achieved notable performance improvements on domain adaptation. However, the effectiveness of these RAG-based architectures remains relatively unexplored when fine-tuned on specialized domains such as customer service for building a reliable conversational AI system. Furthermore, a critical challenge persists in reducing the occurrence of hallucinations while maintaining high domain-specific accuracy. In this paper, we investigated the performance of diverse RAG and RAG-like architectures through domain adaptation and evaluated their ability to generate accurate and relevant response grounded in the contextual knowledge base. To facilitate the evaluation of the models, we constructed a novel dataset HotelConvQA, sourced from wide range of hotel-related conversations and fine-tuned all the models on our domain specific dataset. We also addressed a critical research gap on determining the impact of domain adaptation on reducing hallucinations across different RAG architectures, an aspect that was not properly measured in prior work. Our evaluation shows positive results in all metrics by employing domain adaptation, demonstrating strong performance on QA tasks and providing insights into their efficacy in reducing hallucinations. Our findings clearly indicate that domain adaptation not only enhances the models' performance on QA tasks but also significantly reduces hallucination across all evaluated RAG architectures.

Figures (5)

• Figure 1: Types of questions covered in HotelConvQA. Types were determined by simply taking the first three words of a question. Comparison of which type occurs more are presented here.
• Figure 2: The comparison graph showcases the improvements in retrieval accuracy across all models after being fine-tuned with the hotel domain dataset, as evidenced by notable gains in both Top-5 and Top-20 precision. It also indicates that increasing the number of retrieved documents leads to consistent improvements in retrieval performance which is evident by the parallel rise of Top-5 and Top-20 scores across all variants.
• Figure 3: From the left figure (a) represents the Hallucination Percentage Score of Models after Domain Adaptation. Figure (b) and (c) illustrates the comparison of Hallucination Percentage in Categories across models in Non-Supported and Supported Cases respectively.
• Figure 4: Workflow of Data Preprocessing Stages
• Figure 5: Left: Screenshot of the annotator interface on QA Dataset Generation. Right: Annotator interface for hallucination evaluation and categorization.