Facilitating Instructors-LLM Collaboration for Problem Design in Introductory Programming Classrooms

TL;DR

The paper addresses how instructors can collaborate with LLMs to design introductory programming problems, introducing the INSIGHT Classroom Assistant with an instructor authoring tool and student app. It employs a two-round protocol comparing unguided versus guided LLM interaction to generate problem statements, solutions, and feedback, analyzed through qualitative case studies of three CS1 instructors. Key findings show guided prompting improves efficiency and alignment with learning objectives, while unguided prompts can foster creativity but depend on instructor experience, highlighting a need for a hybrid approach with human oversight. The work offers design guidelines for AI-assisted problem-authoring interfaces and outlines a path for larger-scale validation to balance instructor flexibility with AI assistance, ultimately aiming to enhance programming education workflows.

Abstract

Advancements in Large Language Models (LLMs), such as ChatGPT, offer significant opportunities to enhance instructional support in introductory programming courses. While extensive research has explored the effectiveness of LLMs in supporting student learning, limited studies have examined how these models can assist instructors in designing instructional activities. This work investigates how instructors' expertise in effective activity design can be integrated with LLMs' ability to generate novel and targeted programming problems, facilitating more effective activity creation for programming classrooms. To achieve this, we employ a participatory design approach to develop an instructor-authoring tool that incorporates LLM support, fostering collaboration between instructors and AI in generating programming exercises. This tool also allows instructors to specify common student mistakes and misconceptions, which informs the adaptive feedback generation process. We conduct case studies with three instructors, analyzing how they use our system to design programming problems for their introductory courses. Through these case studies, we assess instructors' perceptions of the usefulness and limitations of LLMs in authoring problem statements for instructional purposes. Additionally, we compare the efficiency, quality, effectiveness, and coverage of designed activities when instructors create problems with and without structured LLM prompting guidelines. Our findings provide insights into the potential of LLMs in enhancing instructor workflows and improving programming education and provide guidelines for designing effective AI-assisted problem-authoring interfaces.

Figures (2)

• Figure 1: Different scenarios in the authoring tool interface and the student app interface in VS Code.
• Figure 2: Example of guided problem generation.