Chatting with a Learning Analytics Dashboard: The Role of Generative AI Literacy on Learner Interaction with Conventional and Scaffolding Chatbots

TL;DR

Investigating the role of GenAI literacy in learner interactions with conventional (reactive) versus scaffolding (proactive) chatbot-assisted LADs shows that while both chatbots significantly improved learner comprehension, those with higher GenAI literacy benefited the most, particularly with conventional chatbots.

Abstract

Learning analytics dashboards (LADs) simplify complex learner data into accessible visualisations, providing actionable insights for educators and students. However, their educational effectiveness has not always matched the sophistication of the technology behind them. Explanatory and interactive LADs, enhanced by generative AI (GenAI) chatbots, hold promise by enabling dynamic, dialogue-based interactions with data visualisations and offering personalised feedback through text. Yet, the effectiveness of these tools may be limited by learners' varying levels of GenAI literacy, a factor that remains underexplored in current research. This study investigates the role of GenAI literacy in learner interactions with conventional (reactive) versus scaffolding (proactive) chatbot-assisted LADs. Through a comparative analysis of 81 participants, we examine how GenAI literacy is associated with learners' ability to interpret complex visualisations and their cognitive processes during interactions with chatbot-assisted LADs. Results show that while both chatbots significantly improved learner comprehension, those with higher GenAI literacy benefited the most, particularly with conventional chatbots, demonstrating diverse prompting strategies. Findings highlight the importance of considering learners' GenAI literacy when integrating GenAI chatbots in LADs and educational technologies. Incorporating scaffolding techniques within GenAI chatbots can be an effective strategy, offering a more guided experience that reduces reliance on learners' GenAI literacy.

Figures (3)

• Figure 1: Three LAD visualisations, representing the activity of two nurses in the first phase of their simulation, used in the current study (from left to right): a bar chart, representing the extent of time dedicated to particular tasks; a communication network, representing the amount of conversation among team members; and a ward map showing location, speech and highest heart rate. Enlarged figures are available in our https://osf.io/mwu3x/?view_only=664e0a7fbdf846cc8471df63923681ac.
• Figure 2: System design of conventional and scaffolding Generative AI (GenAI) chatbots, highlighting four main components: a) distinctive traits separating the conventional chatbot from the scaffolding one, b) interaction dynamics between user prompts, LAD visualisations, and the GenAI agent leveraging retrieval-augmented generation for prompt synthesis, c) a knowledge database with essential contextual information for learning tasks, d) generation of contextually relevant responses using multimodal GenAI. Examples of behaviours: the conventional chatbot (red) focuses on support and explanations, while the scaffolding chatbot (blue) offers guided questions and feedback.
• Figure 3: Comparison plots for the preliminary analysis (left), RQ3 conventional (mid), and scaffolding chatbots (right).