Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Integrating A.I. in Higher Education: Protocol for a Pilot Study with 'SAMCares: An Adaptive Learning Hub'

Syed Hasib Akhter Faruqui, Nazia Tasnim, Iftekhar Ibne Basith, Suleiman Obeidat, Faruk Yildiz

TL;DR

This paper outlines a protocol for a year-long, stratified randomized controlled trial to evaluate SAMCares, an adaptive learning hub that integrates LLaMa-2 70B with a Retrieval-Augmented Generation (RAG) framework to provide context-aware tutoring over SHSU course notes. The study design includes SHSU freshman engineering technology students, a 1:1 randomization to SAMCares or control, and a comprehensive data collection plan incorporating exams, surveys, eye-tracking, and qualitative interviews. The protocol details system-building aspects (RAG integration, 4-bit quantized LLaMa-2 on dual GPUs, secure web interface) and a robust data governance plan with encrypted repositories and de-identified data sharing, aiming to assess improvements in knowledge gains, satisfaction, and cognitive load. If successful, this work could demonstrate the potential of AI-assisted, personalized learning tools to enhance inclusivity and learning outcomes in higher education, while outlining considerations for ethical deployment and scalability.

Abstract

Learning never ends, and there is no age limit to grow yourself. However, the educational landscape may face challenges in effectively catering to students' inclusion and diverse learning needs. These students should have access to state-of-the-art methods for lecture delivery, online resources, and technology needs. However, with all the diverse learning sources, it becomes harder for students to comprehend a large amount of knowledge in a short period of time. Traditional assistive technologies and learning aids often lack the dynamic adaptability required for individualized education plans. Large Language Models (LLM) have been used in language translation, text summarization, and content generation applications. With rapid growth in AI over the past years, AI-powered chatbots and virtual assistants have been developed. This research aims to bridge this gap by introducing an innovative study buddy we will be calling the 'SAMCares'. The system leverages a Large Language Model (LLM) (in our case, LLaMa-2 70B as the base model) and Retriever-Augmented Generation (RAG) to offer real-time, context-aware, and adaptive educational support. The context of the model will be limited to the knowledge base of Sam Houston State University (SHSU) course notes. The LLM component enables a chat-like environment to interact with it to meet the unique learning requirements of each student. For this, we will build a custom web-based GUI. At the same time, RAG enhances real-time information retrieval and text generation, in turn providing more accurate and context-specific assistance. An option to upload additional study materials in the web GUI is added in case additional knowledge support is required. The system's efficacy will be evaluated through controlled trials and iterative feedback mechanisms.

Integrating A.I. in Higher Education: Protocol for a Pilot Study with 'SAMCares: An Adaptive Learning Hub'

TL;DR

This paper outlines a protocol for a year-long, stratified randomized controlled trial to evaluate SAMCares, an adaptive learning hub that integrates LLaMa-2 70B with a Retrieval-Augmented Generation (RAG) framework to provide context-aware tutoring over SHSU course notes. The study design includes SHSU freshman engineering technology students, a 1:1 randomization to SAMCares or control, and a comprehensive data collection plan incorporating exams, surveys, eye-tracking, and qualitative interviews. The protocol details system-building aspects (RAG integration, 4-bit quantized LLaMa-2 on dual GPUs, secure web interface) and a robust data governance plan with encrypted repositories and de-identified data sharing, aiming to assess improvements in knowledge gains, satisfaction, and cognitive load. If successful, this work could demonstrate the potential of AI-assisted, personalized learning tools to enhance inclusivity and learning outcomes in higher education, while outlining considerations for ethical deployment and scalability.

Abstract

Learning never ends, and there is no age limit to grow yourself. However, the educational landscape may face challenges in effectively catering to students' inclusion and diverse learning needs. These students should have access to state-of-the-art methods for lecture delivery, online resources, and technology needs. However, with all the diverse learning sources, it becomes harder for students to comprehend a large amount of knowledge in a short period of time. Traditional assistive technologies and learning aids often lack the dynamic adaptability required for individualized education plans. Large Language Models (LLM) have been used in language translation, text summarization, and content generation applications. With rapid growth in AI over the past years, AI-powered chatbots and virtual assistants have been developed. This research aims to bridge this gap by introducing an innovative study buddy we will be calling the 'SAMCares'. The system leverages a Large Language Model (LLM) (in our case, LLaMa-2 70B as the base model) and Retriever-Augmented Generation (RAG) to offer real-time, context-aware, and adaptive educational support. The context of the model will be limited to the knowledge base of Sam Houston State University (SHSU) course notes. The LLM component enables a chat-like environment to interact with it to meet the unique learning requirements of each student. For this, we will build a custom web-based GUI. At the same time, RAG enhances real-time information retrieval and text generation, in turn providing more accurate and context-specific assistance. An option to upload additional study materials in the web GUI is added in case additional knowledge support is required. The system's efficacy will be evaluated through controlled trials and iterative feedback mechanisms.
Paper Structure (19 sections, 1 equation, 4 figures)

This paper contains 19 sections, 1 equation, 4 figures.

Table of Contents

  1. Introduction
  2. Methods
  3. Study Design and Study Population
  4. Sample Size for the Pilot Study
  5. Recruitment
  6. Randomization
  7. Data Collection
  8. Trial Sessions
  9. Materials and Assessment
  10. Primary and Second Outcome
  11. Data Repository
  12. Documentation and Dissemination Policy
  13. Building SAMCares
  14. Implementing Retrieval-Augmented Generation in SAMCares
  15. Web-Based Interface for Demonstration
  16. ...and 4 more sections

Figures (4)

  • Figure 1: Schematic Representation of Randomization and Data Collection Process for SAMCares Tool Evaluation
  • Figure 2: Retriever-Augmented Generation (RAG) system for SAMCares: Process flowchart for generating context-aware responses using vector embeddings and LLAMA 2 70B model.
  • Figure 3: Interface of SAMCares: An Adaptive Learning Hub that the participants will be using.
  • Figure 4: Different levels of explanation from SAMCares for the same knowledge, the explanations can change depending on the question asked to the tool.