Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

LLMs as Educational Analysts: Transforming Multimodal Data Traces into Actionable Reading Assessment Reports

Eduardo Davalos, Yike Zhang, Namrata Srivastava, Jorge Alberto Salas, Sara McFadden, Sun-Joo Cho, Gautam Biswas, Amanda Goodwin

TL;DR

The paper addresses the gap between outcome-centric reading assessments and the need for actionable insights into student reading behaviors. It combines eye-tracking and other multimodal data with unsupervised clustering to derive reading behavior profiles, which are then translated into teacher-facing reports by specialized LLM agents, with human educators evaluating the outputs. Results demonstrate that LLM-generated reports can be clear, relevant, and useful for instructional planning, particularly when enriched with cluster-based profiles, though formatting and workload considerations remain. The study highlights the value of human-in-the-loop AI in education for delivering scalable, data-driven insights that align with classroom practice and standards, while outlining directions for real-time, interactive decision support and improved user interfaces.

Abstract

Reading assessments are essential for enhancing students' comprehension, yet many EdTech applications focus mainly on outcome-based metrics, providing limited insights into student behavior and cognition. This study investigates the use of multimodal data sources -- including eye-tracking data, learning outcomes, assessment content, and teaching standards -- to derive meaningful reading insights. We employ unsupervised learning techniques to identify distinct reading behavior patterns, and then a large language model (LLM) synthesizes the derived information into actionable reports for educators, streamlining the interpretation process. LLM experts and human educators evaluate these reports for clarity, accuracy, relevance, and pedagogical usefulness. Our findings indicate that LLMs can effectively function as educational analysts, turning diverse data into teacher-friendly insights that are well-received by educators. While promising for automating insight generation, human oversight remains crucial to ensure reliability and fairness. This research advances human-centered AI in education, connecting data-driven analytics with practical classroom applications.

LLMs as Educational Analysts: Transforming Multimodal Data Traces into Actionable Reading Assessment Reports

TL;DR

The paper addresses the gap between outcome-centric reading assessments and the need for actionable insights into student reading behaviors. It combines eye-tracking and other multimodal data with unsupervised clustering to derive reading behavior profiles, which are then translated into teacher-facing reports by specialized LLM agents, with human educators evaluating the outputs. Results demonstrate that LLM-generated reports can be clear, relevant, and useful for instructional planning, particularly when enriched with cluster-based profiles, though formatting and workload considerations remain. The study highlights the value of human-in-the-loop AI in education for delivering scalable, data-driven insights that align with classroom practice and standards, while outlining directions for real-time, interactive decision support and improved user interfaces.

Abstract

Reading assessments are essential for enhancing students' comprehension, yet many EdTech applications focus mainly on outcome-based metrics, providing limited insights into student behavior and cognition. This study investigates the use of multimodal data sources -- including eye-tracking data, learning outcomes, assessment content, and teaching standards -- to derive meaningful reading insights. We employ unsupervised learning techniques to identify distinct reading behavior patterns, and then a large language model (LLM) synthesizes the derived information into actionable reports for educators, streamlining the interpretation process. LLM experts and human educators evaluate these reports for clarity, accuracy, relevance, and pedagogical usefulness. Our findings indicate that LLMs can effectively function as educational analysts, turning diverse data into teacher-friendly insights that are well-received by educators. While promising for automating insight generation, human oversight remains crucial to ensure reliability and fairness. This research advances human-centered AI in education, connecting data-driven analytics with practical classroom applications.

Paper Structure

This paper contains 16 sections, 5 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Eye-Tracking and Multimodal Learning Analytics
  4. Generative AI (GenAI) and Education
  5. Methodology
  6. Study & Participants
  7. Data Preprocessing
  8. Student Grouping via Clustering
  9. LLM Agents: Report Curator and Evaluator
  10. Teacher Survey
  11. Results
  12. Evaluation of Unsupervised Clustering
  13. LLM and Human Evaluation of Assessment Reports
  14. Discussion
  15. Conclusions and Future Work
  16. ...and 1 more sections

Figures (5)

  • Figure 1: Proposed Pipeline for LLM-Driven Assessment Report Generation: By instructing LLMs to role-play as an educational analyst and providing assessment context and data, we construct a prompt that is used to generate a teacher-oriented assessment report.
  • Figure 2: Real-Time AOI Encoding: Performed in-the-moment preprocessing by assigning passage or quiz AOI for each incoming gaze point.
  • Figure 3: K-Means Example Cluster Feature Heatmap: Scaled gaze features are input into an unsupervised clustering algorithm. The cluster centroids are used to construct profiles based on gaze features, resulting in a reading behavior profile for each cluster.
  • Figure 4: LLM-Generated Suffrage Assessment Report
  • Figure 5: LLM and Teacher Evaluations of the Assessment Reports: A complementary evaluation approach, where the LLM provides an global evaluation of report quality, while teachers offer a detailed, section-by-section analysis of its usefulness and clarity.