Integrating Natural Language Prompting Tasks in Introductory Programming Courses

TL;DR

This experience report investigates integrating two natural-language prompting tasks into CS1 to address the traditional bottom-up emphasis on syntax. By embedding EiPE and Prompt Problems across four labs, the study assesses whether these tasks engage different skills than standard programming assessments and how student difficulty relates to performance. Results show a weak correlation between NL prompting performance and traditional assessments, while self-reported difficulty strongly predicts traditional scores but not NL task success, suggesting the prompts promote distinct competencies and broader appeal. Overall, students generally perceive these tasks positively, underscoring their potential to foster AI-literate programming across a diverse student body.

Abstract

Introductory programming courses often emphasize mastering syntax and basic constructs before progressing to more complex and interesting programs. This bottom-up approach can be frustrating for novices, shifting the focus away from problem solving and potentially making computing less appealing to a broad range of students. The rise of generative AI for code production could partially address these issues by fostering new skills via interaction with AI models, including constructing high-level prompts and evaluating code that is automatically generated. In this experience report, we explore the inclusion of two prompt-focused activities in an introductory course, implemented across four labs in a six-week module. The first requires students to solve computational problems by writing natural language prompts, emphasizing problem-solving over syntax. The second involves students crafting prompts to generate code equivalent to provided fragments, to foster an understanding of the relationship between prompts and code. Most of the students in the course had reported finding programming difficult to learn, often citing frustrations with syntax and debugging. We found that self-reported difficulty with learning programming had a strong inverse relationship with performance on traditional programming assessments such as tests and projects, as expected. However, performance on the natural language tasks was less strongly related to self-reported difficulty, suggesting they may target different skills. Learning how to communicate with AI coding models is becoming an important skill, and natural language prompting tasks may appeal to a broad range of students.

Figures (6)

• Figure 1: Natural language prompting tasks requiring the learner to enter a prompt in response to either a code fragment (EiPE) or a visual problem specification (Prompt Problem). The prompt is sent to an LLM, and the resulting code is programmatically evaluated. The EiPE and Prompt Problems were interleaved across Labs 8, 9, 10 and 12.
• Figure 2: An example EiPE problem. The ability to highlight and copy the code is disabled to dissuade students from copying it into ChatGPT.
• Figure 3: Interface layout for a Prompt Problem showing the LeafEater task (Lab 12, Question 3).
• Figure 4: A comparison of student exam score averages for three invigilated exams grouped by difficulty.
• Figure 5: A comparison of students' average success rate on their first attempt to solve the LLM-related questions. The optional Lab 12 was excluded due to low participation.