BlueCodeAgent: A Blue Teaming Agent Enabled by Automated Red Teaming for CodeGen AI

TL;DR

The paper tackles safety risks in CodeGen by proposing BlueCodeAgent, a blue-teaming agent enhanced through automated red teaming. It fuses constitution-based defense with dynamic runtime testing, enabling robust detection of bias instructions, malicious instructions, and vulnerable code across seen and unseen risks. Empirical results show substantial F1 improvements over baselines and safety prompts, with dynamic testing particularly reducing false positives in vulnerability detection. The work demonstrates that comprehensive red-teaming knowledge can power scalable, model-agnostic defenses for code-generation safety and outlines directions for broader risk coverage and multimodal extensions.

Abstract

As large language models (LLMs) are increasingly used for code generation, concerns over the security risks have grown substantially. Early research has primarily focused on red teaming, which aims to uncover and evaluate vulnerabilities and risks of CodeGen models. However, progress on the blue teaming side remains limited, as developing defense requires effective semantic understanding to differentiate the unsafe from the safe. To fill in this gap, we propose BlueCodeAgent, an end-to-end blue teaming agent enabled by automated red teaming. Our framework integrates both sides: red teaming generates diverse risky instances, while the blue teaming agent leverages these to detect previously seen and unseen risk scenarios through constitution and code analysis with agentic integration for multi-level defense. Our evaluation across three representative code-related tasks--bias instruction detection, malicious instruction detection, and vulnerable code detection--shows that BlueCodeAgent achieves significant gains over the base models and safety prompt-based defenses. In particular, for vulnerable code detection tasks, BlueCodeAgent integrates dynamic analysis to effectively reduce false positives, a challenging problem as base models tend to be over-conservative, misclassifying safe code as unsafe. Overall, BlueCodeAgent achieves an average 12.7\% F1 score improvement across four datasets in three tasks, attributed to its ability to summarize actionable constitutions that enhance context-aware risk detection. We demonstrate that the red teaming benefits the blue teaming by continuously identifying new vulnerabilities to enhance defense performance.

Figures (7)

• Figure 1: A case study of BlueCodeAgent on bias instruction detection tasks. For some biased prompts, due to the absence of obvious biased signals, even if concepts such as “biased” are listed in the safety prompt, models usually fail to identify such biased requests. BlueCodeAgent addresses this gap by summarizing constitutions from selected knowledge, using concrete, actionable constraints benefited from red teaming to improve the defense.
• Figure 2: Overview of BlueCodeAgent. BlueCodeAgent is an end-to-end blue teaming framework powered by automated red teaming for code security. By integrating knowledge derived from diverse red teaming and conducting dynamic sandbox-based testing, BlueCodeAgent substantially strengthens the defensive capabilities beyond static LLM analysis.
• Figure 3: F1 scores on BlueCodeEval-Bias.
• Figure 4: F1 scores on BlueCodeEval-Mal(RedCode-based).
• Figure 5: F1 scores on BlueCodeEval-Mal(RMCbench-based).