The Effects of Generative AI Agents and Scaffolding on Enhancing Students' Comprehension of Visual Learning Analytics

TL;DR

Investigating the effectiveness of GenAI agents, particularly when integrated with scaffolding techniques, in improving students' comprehension of VLA suggests that integrating GenAI agents with scaffolding can have lasting positive effects on students' comprehension skills and support genuine learning.

Abstract

Visual learning analytics (VLA) is becoming increasingly adopted in educational technologies and learning analytics dashboards to convey critical insights to students and educators. Yet many students experienced difficulties in comprehending complex VLA due to their limited data visualisation literacy. While conventional scaffolding approaches like data storytelling have shown effectiveness in enhancing students' comprehension of VLA, these approaches remain difficult to scale and adapt to individual learning needs. Generative AI (GenAI) technologies, especially conversational agents, offer potential solutions by providing personalised and dynamic support to enhance students' comprehension of VLA. This study investigates the effectiveness of GenAI agents, particularly when integrated with scaffolding techniques, in improving students' comprehension of VLA. A randomised controlled trial was conducted with 117 higher education students to compare the effects of two types of GenAI agents: passive agents, which respond to student queries, and proactive agents, which utilise scaffolding questions, against standalone scaffolding in a VLA comprehension task. The results show that passive agents yield comparable improvements to standalone scaffolding both during and after the intervention. Notably, proactive GenAI agents significantly enhance students' VLA comprehension compared to both passive agents and standalone scaffolding, with these benefits persisting beyond the intervention. These findings suggest that integrating GenAI agents with scaffolding can have lasting positive effects on students' comprehension skills and support genuine learning.

Figures (10)

• Figure 1: Picture of the healthcare simulation activity with key components labelled and identities masked.
• Figure 2: Visual learning analytics of teamwork in healthcare simulations: a) a bar chart showing students' four prioritisation strategies, b) a social network illustrating communication behaviours among students and other actors, and c) a ward map displaying students' physical positions (hexagon position), verbal communication durations (colour saturation), and peak heart rate locations.
• Figure 3: System design of the passive and proactive generative AI (GenAI) agents, illustrating five main components: A) unique characteristics that differentiate the passive agent from the proactive agent, B) interaction among user prompts, data visualisation, and the GenAI agent for prompt synthesis using retrieval-augmented generation, C) a knowledge database containing essential contextual information for the current task, D) generation of contextually relevant responses using multimodal GenAI, and E) examples of the resulting behaviours of the passive GenAI agent, which focuses on explanation and clarity, and the proactive GenAI agent, which provides scaffolding with guided questions and feedback.
• Figure 4: Example interactions with the two generative AI (GenAI) agents, illustrating the key characteristics of a) the passive GenAI agent and b) the proactive GenAI agent.
• Figure 5: Visual learning analytics infused with data storytelling elements.