StuGPTViz: A Visual Analytics Approach to Understand Student-ChatGPT Interactions

TL;DR

A visual analytics system, StuGPTViz, is presented that tracks and compares temporal patterns in student prompts and the quality of ChatGPT's responses at multiple scales, revealing significant pedagogical insights for instructors.

Abstract

The integration of Large Language Models (LLMs), especially ChatGPT, into education is poised to revolutionize students' learning experiences by introducing innovative conversational learning methodologies. To empower students to fully leverage the capabilities of ChatGPT in educational scenarios, understanding students' interaction patterns with ChatGPT is crucial for instructors. However, this endeavor is challenging due to the absence of datasets focused on student-ChatGPT conversations and the complexities in identifying and analyzing the evolutional interaction patterns within conversations. To address these challenges, we collected conversational data from 48 students interacting with ChatGPT in a master's level data visualization course over one semester. We then developed a coding scheme, grounded in the literature on cognitive levels and thematic analysis, to categorize students' interaction patterns with ChatGPT. Furthermore, we present a visual analytics system, StuGPTViz, that tracks and compares temporal patterns in student prompts and the quality of ChatGPT's responses at multiple scales, revealing significant pedagogical insights for instructors. We validated the system's effectiveness through expert interviews with six data visualization instructors and three case studies. The results confirmed StuGPTViz's capacity to enhance educators' insights into the pedagogical value of ChatGPT. We also discussed the potential research opportunities of applying visual analytics in education and developing AI-driven personalized learning solutions.

Figures (8)

• Figure 1: The summary of task type, count, cognitive level, and a brief description. Sample tasks are provided in the supplementary (A).
• Figure 2: The code schema with revised bloom taxonomy krathwohl2002revision classification.
• Figure 3: The Pattern Summary section of the Pattern View summarizes both between-group and within-group interaction patterns based on the students and tasks selected by the user. Users can click on each grey bar to sort the students according to the selected metric.
• Figure 4: (A) The between group-level background comparison. By default, students are grouped by their background. (B) Under the "Task-Grouping" mode, tasks are grouped by types.
• Figure 5: The Pattern Mining Table within Pattern Nuance. (A) The table headers include pattern length ("L"), interaction pattern ("Pattern"), pattern frequency ("C"), and average score ("Avg."). (B) Example of the "unordered code set" type pattern. Users can click the pattern row to highlight the students utilizing this pattern in (\ref{['fig:TreeDesign']}-A). (C) Example of the "ordered code list" type pattern.