Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Enhanced Question-Answering for Skill-based learning using Knowledge-based AI and Generative AI

Rahul K. Dass, Rochan H. Madhusudhana, Erin C. Deye, Shashank Verma, Timothy A. Bydlon, Grace Brazil, Ashok K. Goel

TL;DR

The paper tackles the difficulty of conveying deep procedural and causal explanations for skill-based learning in online settings. It introduces Ivy, a hybrid agent that combines a knowledge-based TMK representation with Generative AI to produce teleological, causal, and compositional explanations, supported by an end-to-end retrieval and refinement architecture. The authors demonstrate, through human-centered and automated evaluations, that Ivy outperforms a RAG-based baseline in depth, relevance, and fidelity of explanations, validating the TMK framework for explainable skill learning. This work offers a scalable blueprint for delivering structured, domain-aligned explanations in online courses, potentially enhancing learner understanding and problem-solving ability.

Abstract

Supporting learners' understanding of taught skills in online settings is a longstanding challenge. While exercises and chat-based agents can evaluate understanding in limited contexts, this challenge is magnified when learners seek explanations that delve into procedural knowledge (how things are done) and reasoning (why things happen). We hypothesize that an intelligent agent's ability to understand and explain learners' questions about skills can be significantly enhanced using the TMK (Task-Method-Knowledge) model, a Knowledge-based AI framework. We introduce Ivy, an intelligent agent that leverages an LLM and iterative refinement techniques to generate explanations that embody teleological, causal, and compositional principles. Our initial evaluation demonstrates that this approach goes beyond the typical shallow responses produced by an agent with access to unstructured text, thereby substantially improving the depth and relevance of feedback. This can potentially ensure learners develop a comprehensive understanding of skills crucial for effective problem-solving in online environments.

Enhanced Question-Answering for Skill-based learning using Knowledge-based AI and Generative AI

TL;DR

The paper tackles the difficulty of conveying deep procedural and causal explanations for skill-based learning in online settings. It introduces Ivy, a hybrid agent that combines a knowledge-based TMK representation with Generative AI to produce teleological, causal, and compositional explanations, supported by an end-to-end retrieval and refinement architecture. The authors demonstrate, through human-centered and automated evaluations, that Ivy outperforms a RAG-based baseline in depth, relevance, and fidelity of explanations, validating the TMK framework for explainable skill learning. This work offers a scalable blueprint for delivering structured, domain-aligned explanations in online courses, potentially enhancing learner understanding and problem-solving ability.

Abstract

Supporting learners' understanding of taught skills in online settings is a longstanding challenge. While exercises and chat-based agents can evaluate understanding in limited contexts, this challenge is magnified when learners seek explanations that delve into procedural knowledge (how things are done) and reasoning (why things happen). We hypothesize that an intelligent agent's ability to understand and explain learners' questions about skills can be significantly enhanced using the TMK (Task-Method-Knowledge) model, a Knowledge-based AI framework. We introduce Ivy, an intelligent agent that leverages an LLM and iterative refinement techniques to generate explanations that embody teleological, causal, and compositional principles. Our initial evaluation demonstrates that this approach goes beyond the typical shallow responses produced by an agent with access to unstructured text, thereby substantially improving the depth and relevance of feedback. This can potentially ensure learners develop a comprehensive understanding of skills crucial for effective problem-solving in online environments.

Paper Structure

This paper contains 43 sections, 2 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Skill-Based Learning and its Representation in Intelligent Agents
  4. Question-Answering using AI
  5. Generating Reasoned Responses using LLMs
  6. Methodology
  7. Modeling Skills using TMK
  8. TMK Model Development Steps
  9. Generating Reasoned Responses using TMK Models
  10. End-to-End Architecture of Ivy
  11. Knowledge Retrieval Module
  12. Response Generation Module
  13. Response Optimizer Module
  14. Experimental Design and Evaluation Strategies
  15. Developer Perception Evaluation with Baseline model
  16. ...and 28 more sections

Figures (2)

  • Figure 1: High-level TMK model of the 'Partial Order Planning' skill, showing hierarchical problem decomposition.
  • Figure 2: Overall schematic of Ivy's architecture.