Code Shaping: Iterative Code Editing with Free-form AI-Interpreted Sketching

TL;DR

Code shaping introduces a multimodal editing paradigm where programmers annotate code with free-form sketches overlaid on the code and console output to guide AI-driven edits. Through three design stages and multi-study iterations, the work characterizes sketch types, interpretable feedback mechanisms, and gesture-based interfaces that reduce context switching, culminating in an Always-On feedforward interpretation approach. The contributions include a taxonomy of sketches, encounter with interpretation errors and repair strategies, and an end-to-end UI design that merges sketching, coding, and execution into a cohesive workflow, demonstrated via real-use case scenarios. The findings advance practical design principles for integrating sketch-based planning with live code editing, suggesting a viable path to augment programming workflows with embodied, multimodal input while maintaining control and transparency. The work additionally provides open-source tooling for Stage Three and outlines directions for scaling to larger codebases and multi-file edits with dependency tracking and versioned sketches.

Abstract

We introduce the concept of code shaping, an interaction paradigm for editing code using free-form sketch annotations directly on top of the code and console output. To evaluate this concept, we conducted a three-stage design study with 18 different programmers to investigate how sketches can communicate intended code edits to an AI model for interpretation and execution. The results show how different sketches are used, the strategies programmers employ during iterative interactions with AI interpretations, and interaction design principles that support the reconciliation between the code editor and sketches. Finally, we demonstrate the practical application of the code shaping concept with two use case scenarios, illustrating design implications from the study.

Figures (8)

• Figure 1: Interface design from the first stage. (a) Pen tool with color options for code annotation; (b) Canvas tools including select, pan, pen, eraser, and other common shapes; (c) AI-powered "Generate" button for translating sketches to code edits; (d) "Run" button executes Python code and displays output in the console.
• Figure 2: The classification of sketched annotations from participants situated in a quadrant with two spectra, Abstract-Concrete and Procedural-Functional.
• Figure 3: Sketches from stage one showing how participants employ arrows ($\rightarrow$) for different purposes, including command (the intended action of operation), parameter (supplementing the command), and target (the area where the edit should occur); (a) indicating procedural flow between commands; (b) referring to data attributes; (c) modifying a function, with the function as the parameter to supplement the command; (d) applying changes to a target area.
• Figure 4: The interface design from the second stage. (a) Command brushes used to steer AI model's interpretation, including reference, delete, add and replace; (b) interpretation of sketches displayed as tooltips, programmers can click on the preview (recognized sketches) and see the full description of AI's reasoning of actions; (c) programmers can click on the commit button to execute the actions, edited code will be shown in diff view and programmers can accept/reject it.
• Figure 5: The interface design from the third stage. (a) programmers can use one finger tap and drag to select items on canvas; (b) tap longer and drag will select code with contextual menu beside; (c) the always-on feedforward interpretation showing the interpretation of sketches or text, the reasoning of action, and the related code; (d) the gutter will be decorated to indicate which code being referenced and which code will be affected; (e) the related code syntax will be highlight transiently; (f) programmers can commit the changes and (g) draw check/cross to accept/reject code edits.