RedCodeAgent: Automatic Red-teaming Agent against Diverse Code Agents

TL;DR

RedCodeAgent tackles the safety of LLM-based code agents by automating red-teaming with an adaptive memory and a tailored toolbox. It operates in simulated sandbox environments to avoid LLM-based judge biases and is evaluated across multiple programming languages and real-world code assistants. The framework outperforms static jailbreak baselines in attack success rate and rejection rate, and it uncovers vulnerabilities that prior methods miss, demonstrating strong efficiency and adaptability. Collectively, RedCodeAgent offers a scalable, dynamic approach to safety assessment of code agents prior to deployment, with practical implications for securing automated coding workflows.

Abstract

Code agents have gained widespread adoption due to their strong code generation capabilities and integration with code interpreters, enabling dynamic execution, debugging, and interactive programming capabilities. While these advancements have streamlined complex workflows, they have also introduced critical safety and security risks. Current static safety benchmarks and red-teaming tools are inadequate for identifying emerging real-world risky scenarios, as they fail to cover certain boundary conditions, such as the combined effects of different jailbreak tools. In this work, we propose RedCodeAgent, the first automated red-teaming agent designed to systematically uncover vulnerabilities in diverse code agents. With an adaptive memory module, RedCodeAgent can leverage existing jailbreak knowledge, dynamically select the most effective red-teaming tools and tool combinations in a tailored toolbox for a given input query, thus identifying vulnerabilities that might otherwise be overlooked. For reliable evaluation, we develop simulated sandbox environments to additionally evaluate the execution results of code agents, mitigating potential biases of LLM-based judges that only rely on static code. Through extensive evaluations across multiple state-of-the-art code agents, diverse risky scenarios, and various programming languages, RedCodeAgent consistently outperforms existing red-teaming methods, achieving higher attack success rates and lower rejection rates with high efficiency. We further validate RedCodeAgent on real-world code assistants, e.g., Cursor and Codeium, exposing previously unidentified security risks. By automating and optimizing red-teaming processes, RedCodeAgent enables scalable, adaptive, and effective safety assessments of code agents.

Figures (14)

• Figure 1: Illustration of RedCodeAgent on automatic red-teaming against a target code agent. Given a user-provided risk scenario with risk description, RedCodeAgent first conducts a memory search to find the top K most similar successful experiences. Then RedCodeAgent decides whether to query the target code agent (i.e., attack the victim code agent) or call specific tools to optimize the attack. For instance, RedCodeAgent calls GCG from the toolbox and results in the red suffix in the 'Query target code agent' block. After the target code agent responds, an evaluation module will determine whether the current attack is successful. If the attack fails, RedCodeAgent receives feedback from the evaluation and continues the attack. If the attack succeeds, a final reflection is performed, and the successful experience is updated in the memory for future reference. The final output is the successful red-teaming instances.
• Figure 2: RedCodeAgent achieves the highest ASR with comparable time costs.
• Figure 3: The cumulative success and failure rates based on attack trajectory length. The curve of Success Cases shows that RedCodeAgent performs attacks efficiently under short trajectory lengths.
• Figure 4: Average time cost for RedCodeAgent to invoke different tools or query the target code agent in successful cases for each risk scenario. The legend presents the average time required for a single invocation of different tools.
• Figure 5: A case study of RedCodeAgent calling different tools to successfully attack the target code agent. The underscore content shows the impact on the prompt after using the tool.