Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Editrail: Understanding AI Usage by Visualizing Student-AI Interaction in Code

Ashley Ge Zhang, Yan-Ru Jhou, Yinuo Yang, Shamita Rao, Maryam Arab, Yan Chen, Steve Oney

TL;DR

This work tackles the visibility gap in how students use generative AI during programming by introducing Editrail, a timeline-based visualization that integrates AI usage with students' code-edit histories. Through a formative needs-finding study and two Python programming tasks, Editrail is shown to enable instructors to identify AI-enabled patterns more accurately and efficiently than a baseline that relies on code and chat logs alone. A within-subject evaluation with 12 instructors demonstrates improved accuracy (0.79 vs 0.31) and actionable insights for timely interventions, while revealing perceptual gaps in reliance on baseline data. The work also discusses design implications for learning with AI, ethical/privacy considerations, and directions for future work, including aggregated classroom views and broader task coverage. Overall, Editrail provides a concrete pathway for instructors to monitor, assess, and guide AI-assisted programming in a way that aligns with pedagogical goals and policy needs, potentially transforming classroom practice in AI-enabled CS education.

Abstract

Programming instructors have diverse philosophies about integrating generative AI into their classes. Some encourage students to use AI, while others restrict or forbid it. Regardless of their approach, all instructors benefit from understanding how their students actually use AI while writing code. Such insight helps instructors assess whether AI use aligns with their pedagogical goals, enables timely intervention when they find unproductive usage patterns, and establishes effective policies for AI use. However, our survey with programming instructors found that many instructors lack visibility into how students use AI in their code-writing processes. To address this challenge, we introduce Editrail, an interactive system that enables instructors to track students' AI usage, create personalized assessments, and provide timely interventions, all within the workflow of monitoring coding histories. We found that Editrail enables instructors to detect AI use that conflicts with pedagogical goals accurately and to determine when and which students require intervention.

Editrail: Understanding AI Usage by Visualizing Student-AI Interaction in Code

TL;DR

This work tackles the visibility gap in how students use generative AI during programming by introducing Editrail, a timeline-based visualization that integrates AI usage with students' code-edit histories. Through a formative needs-finding study and two Python programming tasks, Editrail is shown to enable instructors to identify AI-enabled patterns more accurately and efficiently than a baseline that relies on code and chat logs alone. A within-subject evaluation with 12 instructors demonstrates improved accuracy (0.79 vs 0.31) and actionable insights for timely interventions, while revealing perceptual gaps in reliance on baseline data. The work also discusses design implications for learning with AI, ethical/privacy considerations, and directions for future work, including aggregated classroom views and broader task coverage. Overall, Editrail provides a concrete pathway for instructors to monitor, assess, and guide AI-assisted programming in a way that aligns with pedagogical goals and policy needs, potentially transforming classroom practice in AI-enabled CS education.

Abstract

Programming instructors have diverse philosophies about integrating generative AI into their classes. Some encourage students to use AI, while others restrict or forbid it. Regardless of their approach, all instructors benefit from understanding how their students actually use AI while writing code. Such insight helps instructors assess whether AI use aligns with their pedagogical goals, enables timely intervention when they find unproductive usage patterns, and establishes effective policies for AI use. However, our survey with programming instructors found that many instructors lack visibility into how students use AI in their code-writing processes. To address this challenge, we introduce Editrail, an interactive system that enables instructors to track students' AI usage, create personalized assessments, and provide timely interventions, all within the workflow of monitoring coding histories. We found that Editrail enables instructors to detect AI use that conflicts with pedagogical goals accurately and to determine when and which students require intervention.
Paper Structure (53 sections, 1 equation, 7 figures, 4 tables)

This paper contains 53 sections, 1 equation, 7 figures, 4 tables.

Table of Contents

  1. INTRODUCTION
  2. RELATED WORK
  3. AI in Programming Education
  4. Understanding Student-AI Interaction in Programming
  5. Providing Feedback based on Students' Progress
  6. NEED FINDING STUDY
  7. Study Method
  8. Participants
  9. Data analysis
  10. Instructors Lack a Clear Understanding of Students' AI Usage.
  11. Tracking Students' AI Usage Is Difficult Due to Process Invisibility and Behavioral Variation
  12. Unclear Tracking of AI Usage Leads to Inconsistent and Unenforceable AI policies
  13. Design Goals to Support Monitoring and Guiding AI Usages
  14. System Design
  15. Editrail's UI Overview
  16. ...and 38 more sections

Figures (7)

  • Figure 1: User interface of Editrail. (1) Timeline View: rows represent code lines and time flows horizontally. Edits are shown as colored overlays indicating different types of AI involvement (d): red for copy–paste, green for autocomplete, and pink for student-typed code that closely resembles AI-generated output. Indicators also show coding progress from start to finish. (2) Code Content View: aligned program text showing the full source code at the selected point in time, enabling instructors to connect timeline annotations with specific code states.
  • Figure 2: Details of Editrail’s timeline interaction. (a) Keystroke-level edit indicators shown along the timeline, where blue ones represent insertion and orange ones represent deletion edits. (b) Zoomed-in timeline view revealing the exact code content associated with individual change indicators for detailed inspection.
  • Figure 3: Details of Editrail's interaction of monitoring AI usage. (a) AI–student dialogue shown at the exact point of interaction. (b) Timeline-to-code projection aligning edits with program text. (c) Precise timing of AI code usage. (d) Code pasted from AI without further edits and code pasted from AI with frequent back-and-forth edits by the student.
  • Figure 4: Three cases of code that originates from AI. (a) the user copies and pastes code directly from an AI agent (e.g., ChatGPT or Copilot chat). (b) the user accepts an autocomplete suggestion (e.g., from Copilot's in-editor hints). (c) The user types code that is very similar to something that was part of their conversation with an AI agent.
  • Figure 5: User interface of Editrail's question creation. (a) Instructors select a point on the coding timeline to create questions. (b) Question type options (multiple choice or open-ended). (c) Code area highlighting AI-contributed lines in student's code. (d) Input field for customizing the question. (e) Button to generate the question. (f) Example of a system-generated question based on the selected code. (g) Generated answer for the question. (h) Send the question to students.
  • ...and 2 more figures