Exploring the Impact of Quizzes Interleaved with Write-Code Tasks in Elementary-Level Visual Programming

TL;DR

A large-scale study with 405 students highlights that the curriculum enhanced with richer quizzes led to higher utility during the post-learning phase and highlights that the curriculum enhanced with richer quizzes led to higher utility during the post-learning phase.

Abstract

We explore the role of quizzes in elementary visual programming domains popularly used for K-8 computing education. Prior work has studied various quiz types, such as fill-in-the-gap write-code questions. However, the overall impact of these quizzes is unclear: studies often show utility in the learning phase when enhanced with quizzes, though limited transfer of utility in the post-learning phase. In this paper, we aim to better understand the impact of different quiz types and whether quizzes focusing on diverse skills (e.g., code debugging and task design) would have higher utility. We design a study with Hour of Code: Maze Challenge by code.org as the base curriculum, interleaved with different quiz types. Specifically, we examine two learning groups: (i) HoC-ACE with diverse quizzes including solution tracing, code debugging, code equivalence, and task design; (ii) HoC-Fill with simple quizzes on solution finding. We conducted a large-scale study with 405 students in grades 6--7. Our results highlight that the curriculum enhanced with richer quizzes led to higher utility during the post-learning phase.

Figures (7)

• Figure 1: Sequence of items in the groups HoC-ACE and HoC-Fill during the learning and post-learning phases. The items in the left gray box are used during the learning phase. In this set, the light blue circles containing "T" represent write-code tasks taken from HoC and the green and orange squares represent quizzes. The items in the right gray box are used during the post-learning phase. In this set, the dark blue circles containing "P" represent write-code tasks taken from PostHoC.
• Figure 2: Distribution of items used in our study based on programming concepts of their solution codes. The quizzes are also classified based on higher-order skills of Bloom's revised taxonomy: analyzing, evaluating, creatinganderson2001taxonomy. See Section \ref{['sec.ourcurr']} for details.
• Figure 3: (a) and (b) show examples of write-code tasks from HoC. (c) and (d) show quizzes based on code debugging and task design from HoC-ACE. (e) and (f) show corresponding quizzes based on solution-finding fill-in-the-gap questions from HoC-Fill.
• Figure 4: istribution of students' grade (year of study) on the left and their self-reported years of programming experience on the right based on the pre-survey data collected. The fraction of students is computed w.r.t. each learning group.
• Figure 5: Performance in the learning phase. (a) shows the fraction of students who scored above a certain threshold on HoC (with maximum score $20$). (b) shows the mean (std err) performance in aggregate and per item category scaled between $0$ and $1$. The "Time (s)" column shows the average seconds spent per item. $^{\textcolor{blue}{\ast\ast}}$ indicates significance with $p < 0.01$ and $^{\textcolor{blue}{\ast}}$ with $p < 0.05$.