CPVis: Evidence-based Multimodal Learning Analytics for Evaluation in Collaborative Programming

TL;DR

CPVis presents an evidence-based multimodal learning analytics system for evaluating collaboration in collaborative programming courses. By collecting multimodal data from real classrooms and employing a flower-based visualization alongside time-based views, CPVis enables simultaneous assessment of group and individual performance, supported by LLM-driven annotations and ENA-based group patterns. A formative study informs design requirements, and a multi-method evaluation (quantitative annotation validation, case studies, and a within-subject user study, N=22) demonstrates that CPVis provides deeper insights, intuitive understanding, and higher instructor confidence compared with baselines. The work advances practical evaluation of collaborative programming at multiple levels and suggests future real-time, narrative, and scalable analytics powered by LLMs. Collectively, CPVis offers a concrete, scalable approach to evidence-based feedback and instructional scaffolding in large auxiliary programming classrooms.

Abstract

As programming education becomes more widespread, many college students from non-computer science backgrounds begin learning programming. Collaborative programming emerges as an effective method for instructors to support novice students in developing coding and teamwork abilities. However, due to limited class time and attention, instructors face challenges in monitoring and evaluating the progress and performance of groups or individuals. To address this issue, we collect multimodal data from real-world settings and develop CPVis, an interactive visual analytics system designed to assess student collaboration dynamically. Specifically, CPVis enables instructors to evaluate both group and individual performance efficiently. CPVis employs a novel flower-based visual encoding to represent performance and provides time-based views to capture the evolution of collaborative behaviors. A within-subject experiment (N=22), comparing CPVis with two baseline systems, reveals that users gain more insights, find the visualization more intuitive, and report increased confidence in their assessments of collaboration.

Figures (12)

• Figure 1: Collaborative programming coding schemes, along with their definitions and examples.
• Figure 2: (a) shows the bar chart of the raw data, (b) presents the results of applying Moving Average Smoothing to reduce anomalies in prediction percentages, and (c) highlights the reduction of visual clutter and emphasizes sequential behavior patterns after merging behaviors of the same category.
• Figure 3: A screenshot of Group 10 view.CPVis applies multimodal learning analysis to provide instructors with evidence for evaluating group and student performance. It consists of three views: Filter View (A) Provides an overview and allows group selection. The selected groups appear in the lasso selection area (A2), and the similarity panel (A3) displays the most similar and different groups based on the search (A1a). Content View (B) Displays group performance, with the B1 panel showing completed codes, the B3a panel illustrating the behavior sequence, and the B3b panel showing student engagement over time. Detail View (C) Presents the group's collaborative programming video (C1) and raw conversation data (C2).
• Figure 4: The iterative design of student glyphs: 1 represents cognitive engagement, and 2 represents behavioral engagement.
• Figure 5: The flower metaphor in CPVis, along with its visual encoding, color coding, and some samples.