AugGen: Augmenting Task-Based Learning in Professional Creative Software with LLM-Generated Scaffolded UIs

TL;DR

AugGen tackles the learning barriers of professional creative software by generating scaffolded, task-aware UIs that surface relevant tools, connect actions to domain concepts, and align with native software workflows. The approach uses an LLM-assisted pipeline to analyze tasks, select tools, generate UI code, and deploy scaffolded interfaces within Blender, with progressive disclosure and concept labeling to support learning. In studies with beginners and experts, scaffolded UIs improved task performance and concept learning for novices while providing valuable qualitative feedback on workflow flexibility and cross-software applicability. This work demonstrates a scalable, educator-friendly method for embedding instructional scaffolds into professional tools, enabling personalized, cross-platform workflows and targeted learning during task execution.

Abstract

Professional creative software often presents steep learning curves due to complex interfaces, lack of structured task-aware guidance, and unfamiliar domain terminology. To address these challenges and augment user learning experience, we introduce AugGen, a method for generating scaffolded user interfaces that simplify interface complexity and support task-based learning. With the user's task, our method surfaces task-relevant tools to reduce distracting features, organizes the tools around task workflow stages to offer execution guidance, connects tools with domain concepts to foster learning engagement, and progressively discloses advanced features to manage learning progress. To evaluate the method, we used our LLM-assisted pipeline to generate two task-specific scaffolded UIs and deployed them in Blender, our professional 3D modeling testbed. We invited both beginner (N=32) and expert (N=8) users to evaluate our implemented interfaces. Results show that the scaffolded interfaces significantly reduced user-perceived task load, enhanced task performance via embedded guidance, and augmented concept learning during task execution.

Figures (11)

• Figure 1: Challenges in professional creative software (C1--C3), prior research to tackle these challenges, and AugGen's focus.
• Figure 2: AI copilots in Blender that offer step-by-step task instructions (left) blender_copilot_oai or automatically execute a task (right) blender_copilot. The copilot's instructions overlook interface cues and operational context. By assuming familiarity with Blender's modes, shortcuts, and domain concepts, the guidance offers an incomplete procedural scaffold for beginner users.
• Figure 3: Technical implementation pipeline built upon the AugGen method. (1) The pipeline takes a user's task description in natural language (e.g., perform UV unwrapping) that informs LLM-assisted task analysis to reason workflow stages and relevant domain concepts (e.g., marking seams, unwrapping, checking and visualization). (2) These workflow stages subsequently guide LLM-supported tool selection and complexity assessment, resulting in a candidate tool list (e.g., tools to mark seams and unwrap UV map). (3) The tool list is fed into LLM-based code generation for the UI implementation. (4) The code is executed to generate a scaffolded interface in professional creative software (e.g., custom panel in Blender). Users (e.g., UI designers, software educators, end users) are allowed to step in to refine LLM-generated artifacts (e.g., workflow, tool list, code) to adjust the scaffolded interface.
• Figure 4: UV unwrapping using the default UI of Blender. Tools are located sparsely and sometimes hidden in keyboard shortcuts or mouse right-clicks. Domain concepts like "seams", "unwrap", and "UV islands" are opaque to users with limited explanation.
• Figure 5: UV unwrapping Basic level of the scaffolded UI.