CodeGuard: Improving LLM Guardrails in CS Education

TL;DR

CodeGuard tackles the gap between general-purpose guardrails and the specific safety needs of CS education by introducing a domain-aligned taxonomy, an 8,000-prompt CodeGuard dataset, and a lightweight guardrail model, PromptShield. The approach includes rigorous annotation and a benchmark showing existing guardrails underperform in this domain, followed by a targeted encoder-based solution that achieves $F_1$ around $0.93$ and preserves educational performance on standard tasks. The work demonstrates that domain-specific guardrails can significantly reduce unsafe or policy-violating code generation with minimal impact on legitimate educational use, and it provides openly available data and models to foster reproducibility. The practical impact lies in enabling safer, more trustworthy AI-assisted CS education and guiding future deployments through instructor-driven taxonomy and ethical considerations.

Abstract

Large language models (LLMs) are increasingly embedded in Computer Science (CS) classrooms to automate code generation, feedback, and assessment. However, their susceptibility to adversarial or ill-intentioned prompts threatens student learning and academic integrity. To cope with this important issue, we evaluate existing off-the-shelf LLMs in handling unsafe and irrelevant prompts within the domain of CS education. We identify important shortcomings in existing LLM guardrails which motivates us to propose CodeGuard, a comprehensive guardrail framework for educational AI systems. CodeGuard includes (i) a first-of-its-kind taxonomy for classifying prompts; (ii) the CodeGuard dataset, a collection of 8,000 prompts spanning the taxonomy; and (iii) PromptShield, a lightweight sentence-encoder model fine-tuned to detect unsafe prompts in real time. Experiments show that PromptShield achieves 0.93 F1 score, surpassing existing guardrail methods. Additionally, further experimentation reveals that CodeGuard reduces potentially harmful or policy-violating code completions by 30-65% without degrading performance on legitimate educational tasks. The code, datasets, and evaluation scripts are made freely available to the community.

Figures (3)

• Figure 1: The taxonomy for coding prompts in academic CS domain.
• Figure 2: Curation of the CodeGuard's dataset.
• Figure 3: Before/after comparison of average Pass@1 (across the four benchmarks in Table \ref{['tab:bench']}) and F1 Score for LLMs fine-tuned on CodeGuard dataset.