PLAID: Supporting Computing Instructors to Identify Domain-Specific Programming Plans at Scale

TL;DR

PLAID presents an instructor-centered, LLM-powered tool to identify domain-specific programming plans in application-focused domains. Through formative interviews, design workshops, and a within-subjects evaluation, the work shows reduced cognitive load and higher productivity when instructors design plans with PLAID versus a baseline. Key contributions include empirical insights into instructor practices, design goals for integrating LLM content into instruction, and a system that enables rapid plan identification and refinement across Pandas, Django, PyTorch, and BeautifulSoup domains. The results support the promise of human-in-the-loop AI for scaling plan-based pedagogies while maintaining pedagogical control and domain relevance.

Abstract

Pedagogical approaches focusing on stereotypical code solutions, known as programming plans, can increase problem-solving ability and motivate diverse learners. However, plan-focused pedagogies are rarely used beyond introductory programming. Our formative study (N=10 educators) showed that identifying plans is a tedious process. To advance plan-focused pedagogies in application-focused domains, we created an LLM-powered pipeline that automates the effortful parts of educators' plan identification process by providing use-case-driven program examples and candidate plans. In design workshops (N=7 educators), we identified design goals to maximize instructors' efficiency in plan identification by optimizing interaction with this LLM-generated content. Our resulting tool, PLAID, enables instructors to access a corpus of relevant programs to inspire plan identification, compare code snippets to assist plan refinement, and facilitates them in structuring code snippets into plans. We evaluated PLAID in a within-subjects user study (N=12 educators) and found that PLAID led to lower cognitive demand and increased productivity compared to the state-of-the-art. Educators found PLAID beneficial for generating instructional material. Thus, our findings suggest that human-in-the-loop approaches hold promise for supporting plan-focused pedagogies at scale.

Figures (10)

• Figure 1: Programming plans that educators have already identified from a variety of programming domains, including (from left to right) web scraping with Beautiful Soup Cunningham_PurposeFirstProgramming_CHI-2021, introductory object-oriented programming Iyer_PatternCensus_SIGCSE-2021, introductory procedural programming Wermelinger_problems-to-programs, and testing with RSpec LojoFox_TestingPlan_ITiCSE-2022.
• Figure 2: An overview of the three-step process to motivate, develop, and evaluate PLAID.
• Figure 3: Illustrations of the three characteristics of the design artifacts, proposed to support instructors' plan identification process by addressing the challenges from our formative study. (A) Instructors can view a vast library of generated programs that achieve a diverse set of tasks in the relevant programming application area to inspire their plan identification; (B) Instructors can compare similar code snippets and other plan components to assist their refinement of plans; and (C) Instructors can follow suggested fields that endorse the structure their final plans should follow.
• Figure 4: Jane's Workflow Diagram. Jane (1) asks the system to suggest a plan; (2) edits the name and goal components of her plan, (3) marks the changeable areas in the plan, (4) browses the list of use case to find code that meets her goals, (5) selects the relevant part of the code from the full program and creates a plan from the selection, (6) switches to the full programs tab to search for specific pieces of code, (7) uses the search bar to search for keywords, and (8) groups plans with similar goals.
• Figure 5: PLAID's reference content is generated through an LLM pipeline that produces output on three levels. First, a wide variety of use cases are generated to create example programs that focus on code's applications. Next, using LLM's explanatory comments that represent subgoals within the code, the examples are segmented into meaningful code snippets. The LLM is queried to generate other plan components for each code snippet. Finally, the code snippets are clustered to identify the most common patterns, representing plan candidates. The full programs are presented in 'Programs' views of PLAID interface, whereas snippets are presented in clusters in the 'Plan Creation' view.