Task Synthesis for Elementary Visual Programming in XLogoOnline Environment

TL;DR

This paper tackles the scarcity of practice tasks in XLogoOnline's Mini level by introducing XLogoSyn, a framework for generating high-quality, difficulty-controlled tasks from a reference task. It employs a three-stage pipeline: (i) template- and SMT-guided generation of code, constraints, and goals with difficulty control; (ii) symbolic execution to craft a compatible grid world and SMT-based element placement; and (iii) a multi-criteria scoring rubric to ensure task quality. Expert evaluation shows XLogoSyn approaches expert-crafted task quality and outperforms baselines, and deployment on XLogoOnline yields meaningful learner benefits, with synthesized tasks elevating next-task success rates. Limitations include rough difficulty alignment, absence of learner-specific personalization, and generation time; future work aims to refine difficulty modeling, personalize task generation, and accelerate synthesis via learned or generative AI techniques.

Abstract

In recent years, the XLogoOnline programming platform has gained popularity among novice learners. It integrates the Logo programming language with visual programming, providing a visual interface for learning computing concepts. However, XLogoOnline offers only a limited set of tasks, which are inadequate for learners to master the computing concepts that require sufficient practice. To address this, we introduce XLogoSyn, a novel technique for synthesizing high-quality tasks for varying difficulty levels. Given a reference task, XLogoSyn can generate practice tasks at varying difficulty levels that cater to the varied needs and abilities of different learners. XLogoSyn achieves this by combining symbolic execution and constraint satisfaction techniques. Our expert study demonstrates the effectiveness of XLogoSyn. We have also deployed synthesized practice tasks into XLogoOnline, highlighting the educational benefits of these synthesized practice tasks.

Figures (4)

• Figure 1: Illustration of XLogoSyn for reference task $87$ from XLOMini xlogoonline. XLogoSyn's input includes a reference task $\texttt{T}^\texttt{in}$, its solution code $\texttt{C}^\texttt{in}$, and the desired difficulty level $\texttt{D}$ for a new practice task. The output includes a generated task $\texttt{T}^\texttt{out}$ and its solution code $\texttt{C}^\texttt{out}$ that satisfies the desired difficulty level w.r.t. the reference task.
• Figure 2: Illustrative examples of reference tasks and their solution codes in XLOMini.
• Figure 3: (a) illustrates the stages of XLogoSyn. (b)--(d) show examples of different components after applying these stages to Fig. \ref{['fig:input-output-example']} (Input). Specifically, (b) shows the input code, its sketch, and the output code, where B1, B2, $\cdots$, B8$\in$ {None, left, right, …} and X$\in$ {2, 3, …}. (c) shows the input code constraints, its sketch, and the output code constraints, where $\texttt{cons\_type}$$\in$ {AtMost, Exactly, StartBy, None} and N$\in$ {1, 2,…}. (d) shows the input goal, its sketch, and the output goal, where $\texttt{task\_type}$$\in$ {Find, FindOnly, FindForbid, ...} and fruit_type$\in$ {strawberry, lemon}.
• Figure 4: The performance of our technique XLogoSyn and three baseline techniques. On the x-axis, we present the aggregated results over all scenarios (All), followed by aggregated results based on task difficulty (Easy, Medium, and Hard) and based on $8$ task types. The y-axis presents the score for overall quality. XLogoSyn demonstrates performance close to ExpertSyn, and surpasses both HumanSyn and RotateFlip.