Evaluation of LLM-based Explanations for a Learning Analytics Dashboard

TL;DR

This study evaluates whether large language model–generated explanations can improve the interpretability and usefulness of learning analytics dashboards. It compares LLM-based explanations against human teacher explanations and no explanations in an expert study with $N=12$ university instructors, using an Open Learner Model dashboard powered by a Performance Factors Analysis model. The findings show that LLM-based explanations and recommendations are significantly more favored, with a large effect size ($d=0.96$), suggesting that LLMs can enhance self-regulated learning experiences while preserving pedagogical standards. The work highlights practical potential for deploying LLM-driven interpretability in digital learning environments, while acknowledging limitations and the need for broader validation.

Abstract

Learning Analytics Dashboards can be a powerful tool to support self-regulated learning in Digital Learning Environments and promote development of meta-cognitive skills, such as reflection. However, their effectiveness can be affected by the interpretability of the data they provide. To assist in the interpretation, we employ a large language model to generate verbal explanations of the data in the dashboard and evaluate it against a standalone dashboard and explanations provided by human teachers in an expert study with university level educators (N=12). We find that the LLM-based explanations of the skill state presented in the dashboard, as well as general recommendations on how to proceed with learning within the course are significantly more favored compared to the other conditions. This indicates that using LLMs for interpretation purposes can enhance the learning experience for learners while maintaining the pedagogical standards approved by teachers.

Figures (1)

• Figure 1: Example of the skill state explanations within the sketch of the dashboard (Condition B).