Enhancing Admission Inquiry Responses with Fine-Tuned Models and Retrieval-Augmented Generation
Aram Virabyan
TL;DR
High-volume university admissions inquiries require fast, accurate, and contextually appropriate responses. The authors propose a hybrid system that combines Retrieval-Augmented Generation with domain-specific fine-tuning, trained on a distilled admissions dataset, implemented via a RAG pipeline with LoRA-enhanced Gemma 9B. Results show the fine-tuned RAG model achieves the highest fact recall (92.7%) and user satisfaction (8.9/10), while RAG alone excels at precise data recall (91.4%), and fine-tuned alone struggles with volatile data. This approach can reduce response times and staff workload in admissions and is extensible to other knowledge-intensive domains such as customer service, law, and healthcare.
Abstract
University admissions offices face the significant challenge of managing high volumes of inquiries efficiently while maintaining response quality, which critically impacts prospective students' perceptions. This paper addresses the issues of response time and information accuracy by proposing an AI system integrating a fine-tuned language model with Retrieval-Augmented Generation (RAG). While RAG effectively retrieves relevant information from large datasets, its performance in narrow, complex domains like university admissions can be limited without adaptation, potentially leading to contextually inadequate responses due to the intricate rules and specific details involved. To overcome this, we fine-tuned the model on a curated dataset specific to admissions processes, enhancing its ability to interpret RAG-provided data accurately and generate domain-relevant outputs. This hybrid approach leverages RAG's ability to access up-to-date information and fine-tuning's capacity to embed nuanced domain understanding. We further explored optimization strategies for the response generation logic, experimenting with settings to balance response quality and speed, aiming for consistently high-quality outputs that meet the specific requirements of admissions communications.