Supporting Students in Navigating LLM-Generated Insecure Code

TL;DR

The paper addresses the risk that LLM-generated code can be insecure and proposes Bifröst, an educational framework that embeds a poisoned LLM into a VS Code workflow to train students to recognize and respond to vulnerabilities. Through a preliminary survey and a classroom deployment with static analysis feedback, the authors show that students become more skeptical of AI-generated code yet retain limited ability to mitigate insecure outputs. Key contributions include the design of a realistic development environment, an adversarial LLM setup, and a vulnerability-feedback pipeline that informs post-surveys. The findings highlight the potential and limitations of security-aware AI-assisted programming education, suggesting that guided learning can increase critical evaluation while underscoring the need for complementary instructional approaches. The practical impact lies in informing instructors how to integrate security considerations into AI-assisted software development curricula.

Abstract

The advent of Artificial Intelligence (AI), particularly large language models (LLMs), has revolutionized software development by enabling developers to specify tasks in natural language and receive corresponding code, boosting productivity. However, this shift also introduces security risks, as LLMs may generate insecure code that can be exploited by adversaries. Current educational approaches emphasize efficiency while overlooking these risks, leaving students underprepared to identify and mitigate security issues in AI-assisted workflows. To address this gap, we present Bifröst, an educational framework that cultivates security awareness in AI-augmented development. Bifröst integrates (1) a Visual Studio Code extension simulating realistic environments, (2) adversarially configured LLMs that generate insecure code, and (3) a feedback system highlighting vulnerabilities. By immersing students in tasks with compromised LLMs and providing targeted security analysis, Bifröst cultivates critical evaluation skills; classroom deployments (n=61) show vulnerability to insecure code, while a post-intervention survey (n=21) indicates increased skepticism toward LLM outputs.

Figures (7)

• Figure 1: Usage Example of Code-generation LLMs.
• Figure 2: Poisoning Attacks in LLMs.
• Figure 3: Overview of Our Framework.
• Figure 4: Example of VS Code Plugin.
• Figure 5: Preliminary Survey Results for Trust Levels.