Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Enhancing LLM-Based Feedback: Insights from Intelligent Tutoring Systems and the Learning Sciences

John Stamper, Ruiwei Xiao, Xinying Hou

TL;DR

The paper addresses the gap in theory-backed design for LLM-based feedback within Intelligent Tutoring Systems by synthesizing historical ITS feedback approaches—expert-generated models, data-driven models, and recent LLM-based methods—and grounding them in learning-science frameworks. It argues that effective LLM feedback requires careful trigger strategies, richer student-context inputs, KLI-aligned content, diverse delivery modalities, and rigorous evaluation beyond traditional metrics. The authors propose a practical, evidence-based design toolkit and guidelines spanning prompting, content selection, deployment, and assessment, aiming to preserve educational integrity and learning gains in the era of generative AI. This work highlights the importance of theoretical grounding to mitigate risks like bias and hallucination while enabling scalable, personalized feedback that aligns with established instructional principles. The practical impact lies in offering a roadmap for researchers and practitioners to develop pedagogically sound, evaluable LLM-powered ITS feedback systems.

Abstract

The field of Artificial Intelligence in Education (AIED) focuses on the intersection of technology, education, and psychology, placing a strong emphasis on supporting learners' needs with compassion and understanding. The growing prominence of Large Language Models (LLMs) has led to the development of scalable solutions within educational settings, including generating different types of feedback in Intelligent Tutoring Systems. However, the approach to utilizing these models often involves directly formulating prompts to solicit specific information, lacking a solid theoretical foundation for prompt construction and empirical assessments of their impact on learning. This work advocates careful and caring AIED research by going through previous research on feedback generation in ITS, with emphasis on the theoretical frameworks they utilized and the efficacy of the corresponding design in empirical evaluations, and then suggesting opportunities to apply these evidence-based principles to the design, experiment, and evaluation phases of LLM-based feedback generation. The main contributions of this paper include: an avocation of applying more cautious, theoretically grounded methods in feedback generation in the era of generative AI; and practical suggestions on theory and evidence-based feedback design for LLM-powered ITS.

Enhancing LLM-Based Feedback: Insights from Intelligent Tutoring Systems and the Learning Sciences

TL;DR

The paper addresses the gap in theory-backed design for LLM-based feedback within Intelligent Tutoring Systems by synthesizing historical ITS feedback approaches—expert-generated models, data-driven models, and recent LLM-based methods—and grounding them in learning-science frameworks. It argues that effective LLM feedback requires careful trigger strategies, richer student-context inputs, KLI-aligned content, diverse delivery modalities, and rigorous evaluation beyond traditional metrics. The authors propose a practical, evidence-based design toolkit and guidelines spanning prompting, content selection, deployment, and assessment, aiming to preserve educational integrity and learning gains in the era of generative AI. This work highlights the importance of theoretical grounding to mitigate risks like bias and hallucination while enabling scalable, personalized feedback that aligns with established instructional principles. The practical impact lies in offering a roadmap for researchers and practitioners to develop pedagogically sound, evaluable LLM-powered ITS feedback systems.

Abstract

The field of Artificial Intelligence in Education (AIED) focuses on the intersection of technology, education, and psychology, placing a strong emphasis on supporting learners' needs with compassion and understanding. The growing prominence of Large Language Models (LLMs) has led to the development of scalable solutions within educational settings, including generating different types of feedback in Intelligent Tutoring Systems. However, the approach to utilizing these models often involves directly formulating prompts to solicit specific information, lacking a solid theoretical foundation for prompt construction and empirical assessments of their impact on learning. This work advocates careful and caring AIED research by going through previous research on feedback generation in ITS, with emphasis on the theoretical frameworks they utilized and the efficacy of the corresponding design in empirical evaluations, and then suggesting opportunities to apply these evidence-based principles to the design, experiment, and evaluation phases of LLM-based feedback generation. The main contributions of this paper include: an avocation of applying more cautious, theoretically grounded methods in feedback generation in the era of generative AI; and practical suggestions on theory and evidence-based feedback design for LLM-powered ITS.
Paper Structure (12 sections, 1 figure, 1 table)

This paper contains 12 sections, 1 figure, 1 table.

Table of Contents

  1. Introduction
  2. The Development of ITS Feedback Generation
  3. Feedback generated from expert-generated learner model
  4. Feedback generated from data-driven learner model
  5. Feedback generated from LLMs
  6. Implications for LLM-based feedback
  7. Trigger to deliver the feedback
  8. Information needed to communicate with LLM as input
  9. Content requested in the generated LLM feedback output
  10. Modality for delivering feedback to students
  11. Evaluate the quality of the generated feedback
  12. Conclusion

Figures (1)

  • Figure 1: Key functions of Intelligent Tutoring Systems from Koedinger et al. Koedinger2013NewPF (left) and Opportunities to Optimize LLM-based Feedback in ITS (right, adapted from Koedinger2013NewPF): 1 - Trigger to deliver the feedback; 2 - Information needed to communicate with LLMs as input; 3 - Content requested in the generated LLM feedback output; 4 - Modality for delivering feedback to students.