Interactions with Prompt Problems: A New Way to Teach Programming with Large Language Models

TL;DR

This paper introduces Prompt Problems, a novel pedagogical approach that uses visual problem representations and prompt engineering to teach programming with large language models. Through the Promptly tool, students craft prompts that generate code via an LLM, which is then verified against test cases to ensure correctness, fostering problem decomposition, reading of AI-generated code, and metacognitive reflection. The authors present exploratory, pilot, and large-scale studies showing how students interact with prompting, the prevalence of iterative strategies, and nuanced perceptions of AI-enabled learning, along with design considerations for scalable educational tooling. Findings suggest that Prompt Problems can supplement traditional coding pedagogy by building prompt-generation skills, exposing students to diverse approaches, and strengthening computational thinking, while also highlighting challenges such as vocabulary gaps, non-determinism, and the need for robust assessment and accessibility considerations. Overall, Promptly demonstrates a promising direction for computing education that leverages AI while maintaining essential competencies in problem understanding and code evaluation.

Abstract

Large Language Models (LLMs) have upended decades of pedagogy in computing education. Students previously learned to code through \textit{writing} many small problems with less emphasis on code reading and comprehension. Recent research has shown that free code generation tools powered by LLMs can solve introductory programming problems presented in natural language with ease. In this paper, we propose a new way to teach programming with Prompt Problems. Students receive a problem visually, indicating how input should be transformed to output, and must translate that to a prompt for an LLM to decipher. The problem is considered correct when the code that is generated by the student prompt can pass all test cases. In this paper we present the design of this tool, discuss student interactions with it as they learn, and provide insights into this new class of programming problems as well as the design tools that integrate LLMs.

Figures (9)

• Figure 1: The tool we created, Promptly, showing the user solving the problem by submitting a prompt.
• Figure 2: Producing a categorization based on age.
• Figure 3: Calculating the average of the "middle" values out of a set of five values (using the metaphor of judges scoring an athletic competition, where the highest and lowest values are excluded).
• Figure 4: The average number of words in each subsequent submission and number of participants that submitted. On the x-axis, 1 is the initial submission (attempt) per question and 2- are subsequent submissions (attempts).
• Figure 5: Lab 10 Exercises 1 (a), 2 (b), 3 (c) and Lab 11 Exercises 1 (d), 2 (e), 3(f)