Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

How Good is ChatGPT in Giving Adaptive Guidance Using Knowledge Graphs in E-Learning Environments?

Patrick Ocheja, Brendan Flanagan, Yiling Dai, Hiroaki Ogata

TL;DR

The paper addresses the challenge of delivering personalized feedback in e-learning by integrating dynamic knowledge graphs with large language models to produce prerequisite-aware, adaptive guidance. It details the AI-sensei framework, including KG construction from Algebra 2 content, question retrieval and ranking, impasse modeling, and a prompting strategy (P1) to tailor LLM responses. Evaluation combines ROUGE-based text similarity and expert judgments, supplemented by a pilot user study, revealing promising but imperfect personalization: easy tasks show high alignment with standard solutions, while harder tasks require more nuanced guidance and carry a higher risk of errors. The work highlights the potential of KG-augmented LLM feedback for AI-driven personalized learning while underscoring the need for human oversight and broader validation across domains, languages, and real-world users.

Abstract

E-learning environments are increasingly harnessing large language models (LLMs) like GPT-3.5 and GPT-4 for tailored educational support. This study introduces an approach that integrates dynamic knowledge graphs with LLMs to offer nuanced student assistance. By evaluating past and ongoing student interactions, the system identifies and appends the most salient learning context to prompts directed at the LLM. Central to this method is the knowledge graph's role in assessing a student's comprehension of topic prerequisites. Depending on the categorized understanding (good, average, or poor), the LLM adjusts its guidance, offering advanced assistance, foundational reviews, or in-depth prerequisite explanations, respectively. Preliminary findings suggest students could benefit from this tiered support, achieving enhanced comprehension and improved task outcomes. However, several issues related to potential errors arising from LLMs were identified, which can potentially mislead students. This highlights the need for human intervention to mitigate these risks. This research aims to advance AI-driven personalized learning while acknowledging the limitations and potential pitfalls, thus guiding future research in technology and data-driven education.

How Good is ChatGPT in Giving Adaptive Guidance Using Knowledge Graphs in E-Learning Environments?

TL;DR

The paper addresses the challenge of delivering personalized feedback in e-learning by integrating dynamic knowledge graphs with large language models to produce prerequisite-aware, adaptive guidance. It details the AI-sensei framework, including KG construction from Algebra 2 content, question retrieval and ranking, impasse modeling, and a prompting strategy (P1) to tailor LLM responses. Evaluation combines ROUGE-based text similarity and expert judgments, supplemented by a pilot user study, revealing promising but imperfect personalization: easy tasks show high alignment with standard solutions, while harder tasks require more nuanced guidance and carry a higher risk of errors. The work highlights the potential of KG-augmented LLM feedback for AI-driven personalized learning while underscoring the need for human oversight and broader validation across domains, languages, and real-world users.

Abstract

E-learning environments are increasingly harnessing large language models (LLMs) like GPT-3.5 and GPT-4 for tailored educational support. This study introduces an approach that integrates dynamic knowledge graphs with LLMs to offer nuanced student assistance. By evaluating past and ongoing student interactions, the system identifies and appends the most salient learning context to prompts directed at the LLM. Central to this method is the knowledge graph's role in assessing a student's comprehension of topic prerequisites. Depending on the categorized understanding (good, average, or poor), the LLM adjusts its guidance, offering advanced assistance, foundational reviews, or in-depth prerequisite explanations, respectively. Preliminary findings suggest students could benefit from this tiered support, achieving enhanced comprehension and improved task outcomes. However, several issues related to potential errors arising from LLMs were identified, which can potentially mislead students. This highlights the need for human intervention to mitigate these risks. This research aims to advance AI-driven personalized learning while acknowledging the limitations and potential pitfalls, thus guiding future research in technology and data-driven education.

Paper Structure

This paper contains 25 sections, 1 equation, 5 figures, 7 tables.

Table of Contents

  1. Introduction
  2. Background
  3. Related Work
  4. Research Methodology
  5. Knowledge Graph Creation, Node Identification and Prerequisites
  6. Question Retrieval and Ranking
  7. Student's Impasse
  8. LLM-based Personalized Solution Generation
  9. Experiment
  10. Evaluation metrics
  11. A pilot user study
  12. Result
  13. Easy (A) question
  14. ROUGE evaluation
  15. Expert evaluation
  16. ...and 10 more sections

Figures (5)

  • Figure 1: System Architecture.
  • Figure 2: Knowledge graph constructed from the textbook Math Algebra 2 by Prentice Hall
  • Figure 3: Experiment setup.
  • Figure 4: Participants' pre-test perspective on use of AI
  • Figure 5: Participants' post-test perspective on use of AI sensei