SPARK: Real-Time Monitoring of Multi-Faceted Programming Exercises

TL;DR

SPARK addresses the challenge of monitoring student progress in multi-faceted programming exercises by offering a checkpoint-based dashboard that supports customizable task groups, automated test suggestions, and real-time and retrospective visualizations. The method combines AI-generated tests, Puppeteer-based interaction simulation, and a component inspector to reveal intermediate outputs and runtime states across many students. Key contributions include a dataset of 22 students solving two web programming tasks, a within-subject user study with 16 instructors validating usefulness, and a scalable design for class-wide monitoring. The work demonstrates that SPARK improves accuracy in diagnosing challenges, increases instructor confidence, and enables targeted, engaging feedback at scale.

Abstract

Monitoring in-class programming exercises can help instructors identify struggling students and common challenges. However, understanding students' progress can be prohibitively difficult, particularly for multi-faceted problems that include multiple steps with complex interdependencies, have no predictable completion order, or involve evaluation criteria that are difficult to summarize across many students (e.g., exercises building interactive web-based user interfaces). We introduce SPARK, a coding exercise monitoring dashboard designed to address these challenges. SPARK allows instructors to flexibly group substeps into checkpoints based on exercise requirements, suggests automated tests for these checkpoints, and generates visualizations to track progress across steps. SPARK also allows instructors to inspect intermediate outputs, providing deeper insights into solution variations. We also construct a dataset of 40-minute keystroke coding data from N=22 learners solving two web programming exercises and provide empirical insights into the perceived usefulness of SPARK through a within-subjects evaluation with 16 programming instructors.

Figures (8)

• Figure 1: Spark consists of five panels. Here's an overview of the Spark dashboard: the Reference Panel (a), which provides instructors with the expected code answer, the webpage, and a DOM tree preview for the programming exercise; the Checkpoints Panel (b), which allows for the creation and display of nested-task checkpoints; the Progress Visualization (c), which presents a visualization of students' programming progress; the Component Inspector (d), which enables instructors to customize inspections of students' output variations; and the My Classroom (e), which contains student code and statistics on overall task performance.
• Figure 2: Checkpoints panel. In the Checkpoints panel, instructors can freely create nested-task checkpoints. Each checkpoint consists of multiple tasks, and each task (a) is made up of two parts: Task Description and Testing Code. Instructors can click the Generate Tests (b) button to view AI-suggested testing code, which can be manually modified. They can then click the Verify Checkpoint (c) button to test the reference code to see if it passes (e) or fails (d).
• Figure 3: My Classroom panel. Each student working on the programming problem has a Student Code Box (e), which includes the student's code and task completion status (a) indicating whether they have passed the task. There is also a Classroom Statistics board (b) showing the class’s performance across different tasks. The My Classroom panel can switch between two modes (c): Real-Time Mode, where instructors can view students' code in real-time, and Timestamp Mode, which records students' code history every minute. In Timestamp Mode, Instructors can use the slider to review the students' code history. If instructors change the order of the student code boxes in the Progress Viz panel or Components Inspector panel, they can use the Reset Order button (d) to revert the Code Boxes to their default order.
• Figure 4: Progress Visualization panel. The Progress Visualization panel includes a diagram (a) that displays students' progress across checkpoints. Each dot along the checkpoint line represents a student's task completion rate for that checkpoint at a particular timestamp (b). Instructors can hover over a dot to highlight a student (c), use the brush tool to select multiple students within that area (d), and click to select an individual student (e), with the selected student remaining highlighted. Additionally, instructors can adjust the slider to view the visualization diagrams at different timestamps (f).
• Figure 5: Components Inspector Panel. The Components Inspector (a) is structured according to checkpoints. Each task is linked to a corresponding inspector board (b) that includes two key features: the Element Inspector (c) and the Element Previewer (e). Instructors can view the inspection results in (d) (a full view of the results is shown in (g)) and preview the element in (f). The inspector automatically simulates interactions before performing the inspection. For example, (h) displays clustered screenshots of the newly added .todoitem following the interaction "Add new item after clicking the add button (Task 4)". Screenshots of identical elements are grouped together to facilitate easier analysis.