Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

LLM-based Multimodal Feedback Produces Equivalent Learning and Better Student Perceptions than Educator Feedback

Chloe Qianhui Zhao, Jie Cao, Jionghao Lin, Kenneth R. Koedinger

TL;DR

This work tackles the challenge of providing timely, scalable, context-aware feedback in digital learning by introducing a real-time LLM-based multimodal feedback system that combines structured textual explanations with visually grounded slide references and streaming audio. Through an online crowdsourcing experiment, the authors compare AI multimodal feedback against fixed educator feedback, measuring learning gains, engagement, and perception. The results show that AI feedback achieves learning outcomes equivalent to educator feedback while outperforming it in perceived clarity, conciseness, specificity, motivation, satisfaction, and cognitive load, with similar levels of correctness, trust, and acceptance. The findings demonstrate the potential for AI-driven multimodal feedback to scale high-quality, adaptive guidance within existing ITS platforms, reducing instructor workload while enhancing student experience and engagement.

Abstract

Providing timely, targeted, and multimodal feedback helps students quickly correct errors, build deep understanding and stay motivated, yet making it at scale remains a challenge. This study introduces a real-time AI-facilitated multimodal feedback system that integrates structured textual explanations with dynamic multimedia resources, including the retrieved most relevant slide page references and streaming AI audio narration. In an online crowdsourcing experiment, we compared this system against fixed business-as-usual feedback by educators across three dimensions: (1) learning effectiveness, (2) learner engagement, (3) perceived feedback quality and value. Results showed that AI multimodal feedback achieved learning gains equivalent to original educator feedback while significantly outperforming it on perceived clarity, specificity, conciseness, motivation, satisfaction, and reducing cognitive load, with comparable correctness, trust, and acceptance. Process logs revealed distinct engagement patterns: for multiple-choice questions, educator feedback encouraged more submissions; for open-ended questions, AI-facilitated targeted suggestions lowered revision barriers and promoted iterative improvement. These findings highlight the potential of AI multimodal feedback to provide scalable, real-time, and context-aware support that both reduces instructor workload and enhances student experience.

LLM-based Multimodal Feedback Produces Equivalent Learning and Better Student Perceptions than Educator Feedback

TL;DR

This work tackles the challenge of providing timely, scalable, context-aware feedback in digital learning by introducing a real-time LLM-based multimodal feedback system that combines structured textual explanations with visually grounded slide references and streaming audio. Through an online crowdsourcing experiment, the authors compare AI multimodal feedback against fixed educator feedback, measuring learning gains, engagement, and perception. The results show that AI feedback achieves learning outcomes equivalent to educator feedback while outperforming it in perceived clarity, conciseness, specificity, motivation, satisfaction, and cognitive load, with similar levels of correctness, trust, and acceptance. The findings demonstrate the potential for AI-driven multimodal feedback to scale high-quality, adaptive guidance within existing ITS platforms, reducing instructor workload while enhancing student experience and engagement.

Abstract

Providing timely, targeted, and multimodal feedback helps students quickly correct errors, build deep understanding and stay motivated, yet making it at scale remains a challenge. This study introduces a real-time AI-facilitated multimodal feedback system that integrates structured textual explanations with dynamic multimedia resources, including the retrieved most relevant slide page references and streaming AI audio narration. In an online crowdsourcing experiment, we compared this system against fixed business-as-usual feedback by educators across three dimensions: (1) learning effectiveness, (2) learner engagement, (3) perceived feedback quality and value. Results showed that AI multimodal feedback achieved learning gains equivalent to original educator feedback while significantly outperforming it on perceived clarity, specificity, conciseness, motivation, satisfaction, and reducing cognitive load, with comparable correctness, trust, and acceptance. Process logs revealed distinct engagement patterns: for multiple-choice questions, educator feedback encouraged more submissions; for open-ended questions, AI-facilitated targeted suggestions lowered revision barriers and promoted iterative improvement. These findings highlight the potential of AI multimodal feedback to provide scalable, real-time, and context-aware support that both reduces instructor workload and enhances student experience.
Paper Structure (36 sections, 3 figures, 2 tables)

This paper contains 36 sections, 3 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Feedback in Digital Learning Environments
  4. Multimodal Feedback: Theories and Practices
  5. AI-Facilitated Multimodal Feedback
  6. The design rationale of AI multimodal feedback system
  7. Learner Interface Design and Integrability
  8. Multimodal Inputs
  9. Multimodal Feedback Composition
  10. Corrective Feedback
  11. Resource Area: Visual and Auditory Supplements
  12. Structured Textual Feedback Generation for Visual Enhancement
  13. RAG-Based Reference Retrieval
  14. Feedback Timeliness Enhancement
  15. Experimental Design
  16. ...and 21 more sections

Figures (3)

  • Figure 1: A screenshot of the learner interface of the AI multimodal feedback system when embedding for an open-ended learning activity. Above the embedded interface: (1) the practice question. On the right side: (2) learner's answer panel. On the left side: (3) visually enhanced AI corrective feedback that guides learners’ understanding: correctness information, highlighted key concepts with their further explanations provided as hover tooltips to keep only text directly relevant to the correction in the main feedback), and actionable advice; and (4) reference materials, including (4a) the most relevant slide page with its slide file link, and (4b) a button to activate AI audio narration for the slide page.
  • Figure 2: Comparison of the feedback composition in different conditions.
  • Figure 3: Pre- and Post-Test Scores (%) Showing Comparable Significant Learning Gains in both Baseline and AI Multimodal Feedback Conditions. Error bars represent ±SEM (Standard Error of the Mean).