AI Teaches the Art of Elegant Coding: Timely, Fair, and Helpful Style Feedback in a Global Course

TL;DR

This paper presents Real-Time Style Feedback (RTSF), an LLM-assisted tool designed to deliver timely, actionable coding-style guidance to CS1 students at scale. Implemented in Code in Place, RTSF analyzes four style facets—Identifier Names, Constants/Magic Numbers, Comments, and Decomposition—via LLMs and static analysis, after students pass functionality tests, and delivers standardized JSON feedback within seconds. In a large randomized control trial with over 8,000 learners, real-time feedback substantially increased engagement (fivefold likelihood of viewing feedback) and students who viewed feedback showed higher style scores and more post-functionality edits, with 79% of such edits incorporating the feedback. The study also critically evaluates safety, bias, and practicality, showing no demonstrated gender bias in feedback and highlighting open-source prompts and code to enable broader adoption and further research in scalable, responsible LLM-based feedback for programming education.

Abstract

Teaching students how to write code that is elegant, reusable, and comprehensible is a fundamental part of CS1 education. However, providing this "style feedback" in a timely manner has proven difficult to scale. In this paper, we present our experience deploying a novel, real-time style feedback tool in Code in Place, a large-scale online CS1 course. Our tool is based on the latest breakthroughs in large-language models (LLMs) and was carefully designed to be safe and helpful for students. We used our Real-Time Style Feedback tool (RTSF) in a class with over 8,000 diverse students from across the globe and ran a randomized control trial to understand its benefits. We show that students who received style feedback in real-time were five times more likely to view and engage with their feedback compared to students who received delayed feedback. Moreover, those who viewed feedback were more likely to make significant style-related edits to their code, with over 79% of these edits directly incorporating their feedback. We also discuss the practicality and dangers of LLM-based tools for feedback, investigating the quality of the feedback generated, LLM limitations, and techniques for consistency, standardization, and safeguarding against demographic bias, all of which are crucial for a tool utilized by students.

Figures (8)

• Figure 1: We employ LLM-based analysis to examine students' code and provide suggestions regarding coding style in real-time. Students can review these suggestions and make edits to improve the style of their code immediately after passing all functionality tests.
• Figure 2: Example Student Program Before Style Feedback
• Figure 3: Example Style Feedback Presentation for the program in Figure 2.
• Figure 4: Students who were nudged and had timely feedback were much more likely to to view style-feedback compared to students in the other experiment groups.
• Figure 5: Feedback viewers improved their style scores more so than non feedback viewers. Z-scores of style score on three assignments throughout the course.