Demystify, Use, Reflect: Preparing students to be informed LLM-users

TL;DR

The paper addresses the problem of students misusing or over-relying on LLMs in CS education by redesigning a post-CS1 course to teach LLM internals, enable iterative verification, and foster reflective practice. It presents a structured approach that includes explicit instruction on how LLMs work, demonstrations, and assignments that require multiple iterations and ethical consideration. The study reports that pre/post surveys showed deeper technical understanding, increased token-level awareness, and a shift toward discerning, collaborative use of LLMs across CS subfields, suggesting improved preparedness for an AI-enabled future. The work demonstrates that opening the 'black box' and embedding verification loops can cultivate responsible, effective LLM use in computing education and can be adapted to other courses.

Abstract

We transitioned our post-CS1 course that introduces various subfields of computer science so that it integrates Large Language Models (LLMs) in a structured, critical, and practical manner. It aims to help students develop the skills needed to engage meaningfully and responsibly with AI. The course now includes explicit instruction on how LLMs work, exposure to current tools, ethical issues, and activities that encourage student reflection on personal use of LLMs as well as the larger evolving landscape of AI-assisted programming. In class, we demonstrate the use and verification of LLM outputs, guide students in the use of LLMs as an ingredient in a larger problem-solving loop, and require students to disclose and acknowledge the nature and extent of LLM assistance. Throughout the course, we discuss risks and benefits of LLMs across CS subfields. In our first iteration of the course, we collected and analyzed data from students pre and post surveys. Student understanding of how LLMs work became more technical, and their verification and use of LLMs shifted to be more discerning and collaborative. These strategies can be used in other courses to prepare students for the AI-integrated future.