Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Comparing the Utility, Preference, and Performance of Course Material Search Functionality and Retrieval-Augmented Generation Large Language Model (RAG-LLM) AI Chatbots in Information-Seeking Tasks

Leonardo Pasquarelli, Charles Koutcheme, Arto Hellas

TL;DR

An LLM-powered AI chatbot is developed that augments the answers that are produced with information from the course materials and highlights that both support mechanisms are seen as useful and that support mechanisms work well for specific tasks, while less so for other tasks.

Abstract

Providing sufficient support for students requires substantial resources, especially considering the growing enrollment numbers. Students need help in a variety of tasks, ranging from information-seeking to requiring support with course assignments. To explore the utility of recent large language models (LLMs) as a support mechanism, we developed an LLM-powered AI chatbot that augments the answers that are produced with information from the course materials. To study the effect of the LLM-powered AI chatbot, we conducted a lab-based user study (N=14), in which the participants worked on tasks from a web software development course. The participants were divided into two groups, where one of the groups first had access to the chatbot and then to a more traditional search functionality, while another group started with the search functionality and was then given the chatbot. We assessed the participants' performance and perceptions towards the chatbot and the search functionality and explored their preferences towards the support functionalities. Our findings highlight that both support mechanisms are seen as useful and that support mechanisms work well for specific tasks, while less so for other tasks. We also observe that students tended to prefer the second support mechanism more, where students who were first given the chatbot tended to prefer the search functionality and vice versa.

Comparing the Utility, Preference, and Performance of Course Material Search Functionality and Retrieval-Augmented Generation Large Language Model (RAG-LLM) AI Chatbots in Information-Seeking Tasks

TL;DR

An LLM-powered AI chatbot is developed that augments the answers that are produced with information from the course materials and highlights that both support mechanisms are seen as useful and that support mechanisms work well for specific tasks, while less so for other tasks.

Abstract

Providing sufficient support for students requires substantial resources, especially considering the growing enrollment numbers. Students need help in a variety of tasks, ranging from information-seeking to requiring support with course assignments. To explore the utility of recent large language models (LLMs) as a support mechanism, we developed an LLM-powered AI chatbot that augments the answers that are produced with information from the course materials. To study the effect of the LLM-powered AI chatbot, we conducted a lab-based user study (N=14), in which the participants worked on tasks from a web software development course. The participants were divided into two groups, where one of the groups first had access to the chatbot and then to a more traditional search functionality, while another group started with the search functionality and was then given the chatbot. We assessed the participants' performance and perceptions towards the chatbot and the search functionality and explored their preferences towards the support functionalities. Our findings highlight that both support mechanisms are seen as useful and that support mechanisms work well for specific tasks, while less so for other tasks. We also observe that students tended to prefer the second support mechanism more, where students who were first given the chatbot tended to prefer the search functionality and vice versa.

Paper Structure

This paper contains 13 sections, 4 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Background
  3. Methodology
  4. Experiment Platform
  5. Study Design
  6. Data Collection
  7. Data Analysis
  8. Ethics Statement
  9. Results
  10. Usefulness and Preference
  11. Task Performance
  12. Discussion
  13. Conclusion

Figures (4)

  • Figure 1: Flow of a user prompt when using the LLM-based chatbot.
  • Figure 2: View of search results when typing "encrypted".
  • Figure 3: Chatbot dialogue with a user query and the chatbot response with links to relevant resources. The user asked what the limits for grading are, to which the chatbot described the grading limits. Clicking on any of the links redirects the user to the corresponding course page.
  • Figure 4: Comparison of performance, broken down by the time to complete an exercise and the received score, grouped by exercise. For every exercise, the left box depicts the search bar and the right box the chatbot.