Drawing Your Programs: Exploring the Applications of Visual-Prompting with GenAI for Teaching and Assessment

TL;DR

This paper argues for visual-prompting via problem decomposition diagrams as a multimodal alternative to text-only prompts in GenAI-assisted programming education. It contends that diagrams reflect authentic design reasoning and can reduce cognitive load, enabling new assessment and learning activities. A case study with 133 students shows GPT-4.1 can generate functional code from student-created diagrams with 81.2% full function extraction and a strong correlation (r=0.776, p<0.001) to human function counts, while also revealing common sources of misinterpretation. The work outlines implications for diagramming guidelines, autograder development, and broader adoption of visual prompting to support scalable, formative assessment in computing education.

Abstract

When designing a program, both novice programmers and seasoned developers alike often sketch out -- or, perhaps more famously, whiteboard -- their ideas. Yet despite the introduction of natively multimodal Generative AI models, work on Human-GenAI collaborative coding has remained overwhelmingly focused on textual prompts -- largely ignoring the visual and spatial representations that programmers naturally use to reason about and communicate their designs. In this proposal and position paper, we argue and provide tentative evidence that this text-centric focus overlooks other forms of prompting GenAI models, such as problem decomposition diagrams functioning as prompts for code generation in their own right enabling new types of programming activities and assessments. To support this position, we present findings from a large introductory Python programming course, where students constructed decomposition diagrams that were used to prompt GPT-4.1 for code generation. We demonstrate that current models are very successful in their ability to generate code from student-constructed diagrams. We conclude by exploring the implications of embracing multimodal prompting for computing education, particularly in the context of assessment.

Figures (4)

• Figure 1: Two examples of program decomposition diagrams collected from the dataset.
• Figure 2: Proportion of successful versus only partially successful function extractions.
• Figure 3: Sources of code deviations and errors linked to diagram interpretation.
• Figure 4: Examples of ambiguous function definitions from student-generated decomposition diagrams.