Automatically Generating Questions About Scratch Programs

TL;DR

This work tackles the challenge of assessing learners' true understanding of Scratch programs by automatically generating Questions about Learner’s Code (QLCs) tailored to Scratch. It extends the LitterBox static analysis tool to produce 30 Scratch-specific QLC types aligned with the Block Model and demonstrates large-scale generation from a dataset of 600,913 projects, yielding 54,118,694 questions. A pilot classroom study with 34 ninth graders shows that students’ ability to answer these questions correlates with their coding performance, supporting QLCs as a meaningful proxy for program comprehension. The approach provides a scalable, automated means for teachers to probe understanding and track learning progress, with open-source tooling and a clear path for integration into learning management systems.

Abstract

When learning to program, students are usually assessed based on the code they wrote. However, the mere completion of a programming task does not guarantee actual comprehension of the underlying concepts. Asking learners questions about the code they wrote has therefore been proposed as a means to assess program comprehension. As creating targeted questions for individual student programs can be tedious and challenging, prior work has proposed to generate such questions automatically. In this paper we generalize this idea to the block-based programming language Scratch. We propose a set of 30 different questions for Scratch code covering an established program comprehension model, and extend the LitterBox static analysis tool to automatically generate corresponding questions for a given Scratch program. On a dataset of 600,913 projects we generated 54,118,694 questions automatically. Our initial experiments with 34 ninth graders demonstrate that this approach can indeed generate meaningful questions for Scratch programs, and we find that the ability of students to answer these questions on their programs relates to their overall performance.

Figures (2)

• Figure 1: Example for the question Which of these blocks controls how many times this loop is executed?
• Figure 2: Correlation between Whisker tests passed and correctly answered QLCs ($r=0.467$, $p=0.005$).