The Right Kind of Help: Evaluating the Effectiveness of Intervention Methods in Elementary-Level Visual Programming

TL;DR

The paper investigates how different automated intervention methods affect elementary students' learning and transfer in visual, block-based programming. Through a large two-phase study with 398 students, it compares code-edit recommendations, code-edit quizzes, and metacognitive planning quizzes against a no-intervention control. All interventions improve learning performance, with quiz-based approaches particularly enhancing transfer to novel tasks and boosting engagement and motivation. The findings highlight a trade-off between cognitive load and cognitive engagement, suggesting designers should favor interactive, reasoning-oriented quizzes to promote durable problem-solving skills. The study also discusses implications for scalable, pedagogically grounded interventions in CS education and points to future work extending to longer exposure and text-based programming environments.

Abstract

Prior work has explored various intervention methods for elementary programming. However, the relative impact of these methods during the learning and post-learning phases remains unclear. In this work, we present a large-scale study comparing the effectiveness of various intervention methods in elementary programming both during learning and on novel tasks post-learning. Specifically, we compare three intervention methods: code-edit recommendations (Code-Rec), quizzes based on code edits (Code-Quiz), and quizzes based on metacognitive strategies (Plan-Quiz), along with a no-intervention control (group None). A total of 398 students (across grades 4-7) participated in a two-phase study: learning phase comprising write-code tasks from the Hour of Code: Maze Challenge with the intervention, followed by a post-learning phase comprising more advanced write-code tasks without any intervention. All intervention methods significantly improved learning performance over the control group while preserving students' problem-solving skills in the post-learning phase. Quiz-based methods further improved performance on novel post-learning tasks. Students in intervention groups also reported greater engagement and perceived skill growth.

Figures (11)

• Figure 1: Illustration of our intervention framework. The interface shows a student working on task T10 during the learning phase. The student can seek the intervention at any time via the "Feedback" button, which links to a specific method. After interacting with the intervention, the student can continue working on the task. The no intervention control does not have "Feedback" button.
• Figure 2: Illustration of the intervention methods evaluated in our study, corresponding to the task T10 and student attempt shown in Figure \ref{['fig1:intro']}. (a) Code-Rec and (b) Code-Quiz presents the intervention as feedback which is adapted to the student's current attempt. (c) Plan-Quiz provide two quizzes for the task, though these quizzes are not adapted to the student's current attempt.
• Figure 3: Task details for the learning and post-learning phases. (a) illustrates the set of write-code tasks in the learning and post-learning phases. (b) presents further details about the tasks based on the programming concepts they cover.
• Figure 4: Illustration of tasks from the learning and post-learning phases. (a) illustrates the most advanced task (T12) in the learning phase, which is also the same as post-learning task P07. (b, c, d) illustrate the novel tasks New$_\texttt{PL}$ from the post-learning phase.
• Figure 5: We present students' self-reported prior programming experience and tools used based on the data collected in the presurvey. We compared prior experience between the intervention groups based on Kruskal-Wallis test of significance kruskal1952use and found no differences. Further details about the participating student population is presented in Section \ref{['sec:method.participant']}.