Probing the Unknown: Exploring Student Interactions with Probeable Problems at Scale in Introductory Programming

TL;DR

The paper investigates Probeable Problems, automatically gradable tasks with deliberately vague specifications, as a means to train ambiguity elicitation and metacognitive skills in introductory programming. Using a large-scale deployment (P7–P9 labs) with CodeRunner, the study combines quantitative analysis of over 2,800 attempts and 44,000 probes with a qualitative thematic analysis of student responses to probing. Key findings show that probing before coding correlates with higher course performance, that higher-performing students rely more on inexpensive, pre-coding clarifications, and that students perceive real-world relevance and learning benefits despite some frustration and a preference for traditional tasks. The work demonstrates that Probeable Problems can be scaled in large courses to foster critical thinking and problem-solving strategies, offering a practical approach to mitigate issues arising from ambiguous requirements and AI-assisted programming.

Abstract

Introductory programming courses often rely on small code-writing exercises that have clearly specified problem statements. This limits opportunities for students to practice how to clarify ambiguous requirements -- a critical skill in real-world programming. In addition, the emerging capabilities of large language models (LLMs) to produce code from well-defined specifications may harm student engagement with traditional programming exercises. This study explores the use of ``Probeable Problems'', automatically gradable tasks that have deliberately vague or incomplete specifications. Such problems require students to submit test inputs, or `probes', to clarify requirements before implementation. Through analysis of over 40,000 probes in an introductory course, we identify patterns linking probing behaviors to task success. Systematic strategies, such as thoroughly exploring expected behavior before coding, resulted in fewer incorrect code submissions and correlated with course success. Feedback from nearly 1,000 participants highlighted the challenges and real-world relevance of these tasks, as well as benefits to critical thinking and metacognitive skills. Probeable Problems are easy to set up and deploy at scale, and help students recognize and resolve uncertainties in programming problems.

Figures (3)

• Figure 1: The CodeRunner platform, a widely used automated grading tool, configured to accept probes for P7: "Implement a function to count the number of integers between a and b in an array of length n". Students modify inputs in the upper editing pane, and view the expected output in the lower pane.
• Figure 2: The cumulative percentage of students ($y$-axis) in each grade category for each Probeable Problem as a function of the number of probes ($x$-axis) submitted prior to the first code submission ($F$ or $S$).
• Figure 3: Box plots showing the ratio of probes issued per code submission in each attempt, across all grade categories and problems. We exclude outliers where this ratio exceeds 35 (i.e., 2.9%, 1.3%, and 0.7% of P7, P8, and P9 attempts respectively).