Exploring the Role of Tracing in AI-Supported Planning for Algorithmic Reasoning

TL;DR

This work investigates whether adding explicit execution traces to AI-supported planning shifts learners' algorithmic reasoning from code-like descriptions toward goal-directed representations. In a controlled experiment with 20 students, trace-based planning produced more compact, concept-focused plans and slightly better partial correctness, but did not significantly improve final coding performance or AI feedback quality. The findings suggest tradeoffs between richer representational structure and translation difficulty to code, informing design guidelines for integrating natural language, tracing, and coding in AI-assisted programming tools. The study highlights how traces can illuminate learners' reasoning while preserving the need for effective, reliable AI guidance in educational settings.

Abstract

AI-powered planning tools show promise in supporting programming learners by enabling early, formative feedback on their thinking processes prior to coding. To date, however, most AI-supported planning tools rely on students' natural-language explanations, using LLMs to interpret learners' descriptions of their algorithmic intent. Prior to the emergence of LLM-based systems, CS education research extensively studied trace-based planning in pen-and-paper settings, demonstrating that reasoning through stepwise execution with explicit state transitions helps learners build and refine mental models of program behavior. Despite its potential, little is known about how tracing interacts with AI-mediated feedback and whether integrating tracing into AI-supported planning tools leads to different learning processes or interaction dynamics compared to natural-language-based planning alone. We study how requiring learners to produce explicit execution traces with an AI-supported planning tool affects their algorithmic reasoning. In a between-subjects study with 20 students, tracing shifted learners away from code-like, line-by-line descriptions toward more goal-driven reasoning about program behavior. Moreover, it led to more consistent partially correct solutions, although final coding performance remained comparable across conditions. Notably, tracing did not significantly affect the quality or reliability of LLM-generated feedback. These findings reveal tradeoffs in combining tracing with AI-supported planning and inform design guidelines for integrating natural language, tracing, and coding to support iterative reasoning throughout the programming process.

Figures (5)

• Figure 1: Left: Natural-language–based planning interface, in which students decompose their solution into structured steps and optional sub-steps (indented) using free-form textual explanations. Right: Trace-based planning interface, where students articulate free-form textual explanations and additionally construct a step-by-step execution trace showing the progression of predefined variables on a given input. In both conditions, learners iteratively refine their plan using feedback from LLMs prior to coding.
• Figure 2: Study workflow. Participants completed a greedy algorithm task under either a trace-based or natural-language-based planning condition, both supported with AI-generated feedback. After iteratively refining their plan, students proceeded to implement their solution in code, without AI help.
• Figure 3: Comparison of (a) step count (b) frequency of control-flow references (c) plan-derived code outcomes (d) student code performance (e) semantic alignment between plan-derived and student code (cosine similarity between code embeddings) across conditions. Error bars represent standard deviation. Statistically significant differences were observed for step count and control-flow references ($p < 0.05$; indicated by $*$), whereas performance and alignment differences were not statistically significant.
• Figure 4: For each participant, cosine similarity was computed between consecutive submissions $(t, t+1)$ in chronological order. For each submission index $t$, the plotted value represents the mean similarity across all participants who produced at least $t+1$ submissions; error bars indicate standard deviation. Participants contributed varying numbers of submissions; thus, later indices reflect fewer observations.
• Figure 5: Distribution of self-reported learner perceptions across conditions. Each bar represents the number of responses on a 7-point Likert scale (1 = Strongly Disagree, 7 = Strongly Agree) for each survey item. $*$ indicates significance level at $p<0.05$ (Mann-Whitney U).