Exploring the Design Space of Cognitive Engagement Techniques with AI-Generated Code for Enhanced Learning

TL;DR

This paper conducts a systematic design exploration to develop seven cognitive engagement techniques aimed at promoting deeper engagement with AI-generated code, highlighting the most effective technique: guiding learners through the step-by-step problem-solving process.

Abstract

Novice programmers are increasingly relying on Large Language Models (LLMs) to generate code for learning programming concepts. However, this interaction can lead to superficial engagement, giving learners an illusion of learning and hindering skill development. To address this issue, we conducted a systematic design exploration to develop seven cognitive engagement techniques aimed at promoting deeper engagement with AI-generated code. In this paper, we describe our design process, the initial seven techniques and results from a between-subjects study (N=82). We then iteratively refined the top techniques and further evaluated them through a within-subjects study (N=42). We evaluate the friction each technique introduces, their effectiveness in helping learners apply concepts to isomorphic tasks without AI assistance, and their success in aligning learners' perceived and actual coding abilities. Ultimately, our results highlight the most effective technique: guiding learners through the step-by-step problem-solving process, where they engage in an interactive dialog with the AI, prompting what needs to be done at each stage before the corresponding code is revealed.

Figures (5)

• Figure 1: Summary of cognitive engagement techniques mapped within the design space. The first set of columns (in orange) represents learning opportunities aligned with Bloom's taxonomy. The second set (in blue) corresponds to cognitive engagement levels based on the ICAP framework. The final two columns (in purple) reflect whether the technique either reveals the code before requiring user engagement or requires user engagement before revealing the solution.
• Figure 2: Overview of Study 1 procedure: Participants began with a pre-test, were split into 8 balanced groups, trained with AI using their assigned cognitive engagement technique, completed post-training questionnaires, took a short break, and then performed manual coding tasks for evaluation.
• Figure 3: An overview of results from Study 1 (left to right): the performance on the two manual coding tasks. Average time spent on each technique during the training. Perceived understanding and reported willingness to use each technique (on 5-point Likert scales).
• Figure 4: Cognitive load results for 8 techniques. Each box-plot displays the six dimensions of the raw NASA TLX (mental, physical, temporal demand, inverted performance, effort, and frustration) reported by participants after using each the techniques for four coding tasks. Each technique is represented by a different color, with higher values indicating higher task load.
• Figure 5: Study 2 Procedure Overview: Participants used each technique to learn complex coding tasks about three topics: queue, stack, and double-ended queue (deque). Each topic involved four steps: a short video tutorial (2 min), a warm-up task (5 min), a main task (15 min), and a questionnaire (3 min). After a 15-minute break, participants complete two evaluation tasks (10 min each) for each technique without AI assistance. Techniques during training were fully counterbalanced, while the topics had the same order in both training and evaluation phases of the experiment.