A Survey of LLM-Based Applications in Programming Education: Balancing Automation and Human Oversight

TL;DR

This survey addresses the challenge of providing scalable, personalized programming-education support amid TA shortages by examining how large language models (LLMs) are applied in three areas: formative code feedback, assessment, and knowledge modeling. It identifies design patterns that blend automated LLM outputs with educator oversight, scaffolding, and evaluation, and finds that hybrid, human-in-the-loop approaches tend to be more pedagogically effective than fully automated solutions. The work highlights opportunities to enhance transparency, pedagogy alignment, and adaptability to diverse learning contexts, while noting limitations in current evaluations and the need for course-specific tuning. Overall, the paper advocates for systems that augment rather than replace human expertise, guiding future development toward pedagogy-informed, context-aware LLM-enabled educational tools.

Abstract

Novice programmers benefit from timely, personalized support that addresses individual learning gaps, yet the availability of instructors and teaching assistants is inherently limited. Large language models (LLMs) present opportunities to scale such support, though their effectiveness depends on how well technical capabilities are aligned with pedagogical goals. This survey synthesizes recent work on LLM applications in programming education across three focal areas: formative code feedback, assessment, and knowledge modeling. We identify recurring design patterns in how these tools are applied and find that interventions are most effective when educator expertise complements model output through human-in-the-loop oversight, scaffolding, and evaluation. Fully automated approaches are often constrained in capturing the pedagogical nuances of programming education, although human-in-the-loop designs and course specific adaptation offer promising directions for future improvement. Future research should focus on improving transparency, strengthening alignment with pedagogy, and developing systems that flexibly adapt to the needs of varied learning contexts.