Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Pedagogy-driven Evaluation of Generative AI-powered Intelligent Tutoring Systems

Kaushal Kumar Maurya, Ekaterina Kochmar

TL;DR

This paper surveys the evaluation landscape for GenAI-powered Intelligent Tutoring Systems (ITS) within AIED and argues that reliable, pedagogy-driven benchmarks are missing, hindering progress. It analyzes traditional, NLG-based, and pedagogically oriented evaluation approaches, highlighting gaps in capturing learning science principles and the pedagogical value of tutoring. The authors propose three concrete research directions—Evaluation Unification (taxonomy and MRBench), Measuring Pedagogical Guidance (metacognitive-aligned guidance), and Measuring Active Learning (graph-based dialogue modeling with GNNs and related architectures)—to create unified, scalable, and pedagogically sound evaluation methodologies. By combining standardized frameworks with real-world experimentation, the work aims to enable reliable assessment of ITS effectiveness and broader educational impact of GenAI tutors.

Abstract

The interdisciplinary research domain of Artificial Intelligence in Education (AIED) has a long history of developing Intelligent Tutoring Systems (ITSs) by integrating insights from technological advancements, educational theories, and cognitive psychology. The remarkable success of generative AI (GenAI) models has accelerated the development of large language model (LLM)-powered ITSs, which have potential to imitate human-like, pedagogically rich, and cognitively demanding tutoring. However, the progress and impact of these systems remain largely untraceable due to the absence of reliable, universally accepted, and pedagogy-driven evaluation frameworks and benchmarks. Most existing educational dialogue-based ITS evaluations rely on subjective protocols and non-standardized benchmarks, leading to inconsistencies and limited generalizability. In this work, we take a step back from mainstream ITS development and provide comprehensive state-of-the-art evaluation practices, highlighting associated challenges through real-world case studies from careful and caring AIED research. Finally, building on insights from previous interdisciplinary AIED research, we propose three practical, feasible, and theoretically grounded research directions, rooted in learning science principles and aimed at establishing fair, unified, and scalable evaluation methodologies for ITSs.

Pedagogy-driven Evaluation of Generative AI-powered Intelligent Tutoring Systems

TL;DR

This paper surveys the evaluation landscape for GenAI-powered Intelligent Tutoring Systems (ITS) within AIED and argues that reliable, pedagogy-driven benchmarks are missing, hindering progress. It analyzes traditional, NLG-based, and pedagogically oriented evaluation approaches, highlighting gaps in capturing learning science principles and the pedagogical value of tutoring. The authors propose three concrete research directions—Evaluation Unification (taxonomy and MRBench), Measuring Pedagogical Guidance (metacognitive-aligned guidance), and Measuring Active Learning (graph-based dialogue modeling with GNNs and related architectures)—to create unified, scalable, and pedagogically sound evaluation methodologies. By combining standardized frameworks with real-world experimentation, the work aims to enable reliable assessment of ITS effectiveness and broader educational impact of GenAI tutors.

Abstract

The interdisciplinary research domain of Artificial Intelligence in Education (AIED) has a long history of developing Intelligent Tutoring Systems (ITSs) by integrating insights from technological advancements, educational theories, and cognitive psychology. The remarkable success of generative AI (GenAI) models has accelerated the development of large language model (LLM)-powered ITSs, which have potential to imitate human-like, pedagogically rich, and cognitively demanding tutoring. However, the progress and impact of these systems remain largely untraceable due to the absence of reliable, universally accepted, and pedagogy-driven evaluation frameworks and benchmarks. Most existing educational dialogue-based ITS evaluations rely on subjective protocols and non-standardized benchmarks, leading to inconsistencies and limited generalizability. In this work, we take a step back from mainstream ITS development and provide comprehensive state-of-the-art evaluation practices, highlighting associated challenges through real-world case studies from careful and caring AIED research. Finally, building on insights from previous interdisciplinary AIED research, we propose three practical, feasible, and theoretically grounded research directions, rooted in learning science principles and aimed at establishing fair, unified, and scalable evaluation methodologies for ITSs.
Paper Structure (15 sections, 1 figure, 1 table)

This paper contains 15 sections, 1 figure, 1 table.

Table of Contents

  1. Introduction
  2. Current State of Evaluation for ITS
  3. Traditional Evaluation Approaches
  4. NLG-based Evaluation
  5. Pedagogically Oriented Evaluation
  6. Key Challenges in Evaluation of ITS
  7. Challenge #1: Diverse Pragmatics of Evaluation
  8. Challenge #1: Diverse Pragmatics of Evaluation
  9. Challenge #2: Lack of Evaluation Unification
  10. Path Forward
  11. Research Line #1: Evaluation Unification
  12. Research Line #2: Measuring Pedagogical Guidance
  13. Research Line #3: Measuring Active Learning
  14. Implications of Proposed Research Directions
  15. Conclusion

Figures (1)

  • Figure 1: (Left) Flowchart illustrating good and bad tutor responses to measure pedagogical guidance. (Right) Conversation flow graph to measure the student's active learning, where nodes represent different utterance features and edge weights indicate the intensity of conversational dependency.