Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

From Questions to Insightful Answers: Building an Informed Chatbot for University Resources

Subash Neupane, Elias Hossain, Jason Keith, Himanshu Tripathi, Farbod Ghiasi, Noorbakhsh Amiri Golilarz, Amin Amirlatifi, Sudip Mittal, Shahram Rahimi

TL;DR

This work introduces BarkPlug v.2, an LLM-based chatbot for university resources that uses a Retrieval Augmented Generation pipeline to ground answers in a university-wide external corpus. The system vectorizes 42 MSU campus resources with text embeddings, stores them in Chroma DB, and retrieves relevant chunks via LangChain before generating responses with GPT-3.5-turbo, enabling domain-specific, interactive information access. Evaluation with the RAGAS framework yields high grounding and relevance (average ~0.97 across categories), while a System Usability Scale (SUS) assessment reports a practical mean of 67.75 from 50 students, indicating solid usability with room for improvements. The work discusses implementation details, dataset construction, limitations (e.g., no ASR, multilingual support) and future directions (multilingual support, ASR integration, improved retrieval strategies), highlighting BarkPlug v.2’s potential for broader deployment across universities and sectors that require structured, searchable campus information.

Abstract

This paper presents BARKPLUG V.2, a Large Language Model (LLM)-based chatbot system built using Retrieval Augmented Generation (RAG) pipelines to enhance the user experience and access to information within academic settings.The objective of BARKPLUG V.2 is to provide information to users about various campus resources, including academic departments, programs, campus facilities, and student resources at a university setting in an interactive fashion. Our system leverages university data as an external data corpus and ingests it into our RAG pipelines for domain-specific question-answering tasks. We evaluate the effectiveness of our system in generating accurate and pertinent responses for Mississippi State University, as a case study, using quantitative measures, employing frameworks such as Retrieval Augmented Generation Assessment(RAGAS). Furthermore, we evaluate the usability of this system via subjective satisfaction surveys using the System Usability Scale (SUS). Our system demonstrates impressive quantitative performance, with a mean RAGAS score of 0.96, and experience, as validated by usability assessments.

From Questions to Insightful Answers: Building an Informed Chatbot for University Resources

TL;DR

This work introduces BarkPlug v.2, an LLM-based chatbot for university resources that uses a Retrieval Augmented Generation pipeline to ground answers in a university-wide external corpus. The system vectorizes 42 MSU campus resources with text embeddings, stores them in Chroma DB, and retrieves relevant chunks via LangChain before generating responses with GPT-3.5-turbo, enabling domain-specific, interactive information access. Evaluation with the RAGAS framework yields high grounding and relevance (average ~0.97 across categories), while a System Usability Scale (SUS) assessment reports a practical mean of 67.75 from 50 students, indicating solid usability with room for improvements. The work discusses implementation details, dataset construction, limitations (e.g., no ASR, multilingual support) and future directions (multilingual support, ASR integration, improved retrieval strategies), highlighting BarkPlug v.2’s potential for broader deployment across universities and sectors that require structured, searchable campus information.

Abstract

This paper presents BARKPLUG V.2, a Large Language Model (LLM)-based chatbot system built using Retrieval Augmented Generation (RAG) pipelines to enhance the user experience and access to information within academic settings.The objective of BARKPLUG V.2 is to provide information to users about various campus resources, including academic departments, programs, campus facilities, and student resources at a university setting in an interactive fashion. Our system leverages university data as an external data corpus and ingests it into our RAG pipelines for domain-specific question-answering tasks. We evaluate the effectiveness of our system in generating accurate and pertinent responses for Mississippi State University, as a case study, using quantitative measures, employing frameworks such as Retrieval Augmented Generation Assessment(RAGAS). Furthermore, we evaluate the usability of this system via subjective satisfaction surveys using the System Usability Scale (SUS). Our system demonstrates impressive quantitative performance, with a mean RAGAS score of 0.96, and experience, as validated by usability assessments.
Paper Structure (16 sections, 5 figures, 2 tables)

This paper contains 16 sections, 5 figures, 2 tables.

Table of Contents

  1. Introduction
  2. BACKGROUND AND RELATED WORK
  3. Large Language Models (LLMs)
  4. Retrieval Augmented Generation (RAG)
  5. Related Works
  6. BarkPlug v.2 Architecture & Methodology
  7. Context Retrieval
  8. Completion
  9. Experiments and Results
  10. Dataset Description & Preparation
  11. Quantitative Evaluation
  12. Usability Assessment
  13. Detailed Analysis and Insights
  14. Implementation Detail
  15. Limitations & Future Direction
  16. ...and 1 more sections

Figures (5)

  • Figure 1: Comparative example of completion (response generation) without using the RAG approach versus using the RAG approach for a given user prompt related to specific individual at Mississippi State University.
  • Figure 2: Overview of BarkPlug v.2's two phase architecture. The first phase Context retrieval is responsible to retrieve relevant documents based on the user prompt. The second phase, Completion responsible of generating personalized responses utlilizing retrieved documents as context along with user prompt.
  • Figure 3: Similarity score threshold retrieval.
  • Figure 4: A graphical depiction of response generation. Red highlights indicate BarkPlug v.2's ability to contextualize and generate factually grounded response.
  • Figure 5: Distribution of average System Usability Scale (SUS) scores.