Dynamic Risk Assessments for Offensive Cybersecurity Agents

TL;DR

This work investigates the dynamic risk posed by autonomous offensive cybersecurity agents that can self-improve under a fixed compute budget. It argues that cybersecurity tasks offer strong verifiers and high incentives, making adversaries likely to explore self-improvement strategies, especially across five defined degrees of freedom. Through experiments on three Capture-the-Flag benchmarks, the authors show that modest compute (e.g., 8 H100 GPU hours) can yield substantial performance gains, up to over 40% relative to baselines, with iterative prompt refinement repeatedly emerging as a high-impact but efficient risk pathway. The study advocates for dynamic risk assessments that account for both deployment-time and adaptation-time compute and for policy and safety considerations that reflect this expanded threat model.

Abstract

Foundation models are increasingly becoming better autonomous programmers, raising the prospect that they could also automate dangerous offensive cyber-operations. Current frontier model audits probe the cybersecurity risks of such agents, but most fail to account for the degrees of freedom available to adversaries in the real world. In particular, with strong verifiers and financial incentives, agents for offensive cybersecurity are amenable to iterative improvement by would-be adversaries. We argue that assessments should take into account an expanded threat model in the context of cybersecurity, emphasizing the varying degrees of freedom that an adversary may possess in stateful and non-stateful environments within a fixed compute budget. We show that even with a relatively small compute budget (8 H100 GPU Hours in our study), adversaries can improve an agent's cybersecurity capability on InterCode CTF by more than 40\% relative to the baseline -- without any external assistance. These results highlight the need to evaluate agents' cybersecurity risk in a dynamic manner, painting a more representative picture of risk.

Figures (13)

• Figure 1: (a) We introduce a new threat model in which adversaries will have at least five degrees of freedom to modify offensive cybersecurity agents for improved performance. (b) Under this threat model, we assess the risk of offensive cybersecurity agents by dynamically analyzing how far adversaries can push along each axis on InterCode CTF (Test), within a fixed 8 H100 GPU Hours compute budget.
• Figure 2: Increasing the number of repeated samples $k$ and max rounds of interactions $N$ will significantly improve the accuracy, though the rate of improvement slows due to diminishing returns.
• Figure 3: Iterative prompt refinement can help the agent to search more efficiently, resulting in higher pass@$k$ scores compared to repeated sampling.
• Figure 4: Self-Training shows in-domain generalization, even without a large amount of data or external assistance. However, it comes with trade-offs in generation diversity, especially when the model is fine-tuned for more epochs.
• Figure 5: Even using the same core model for the meta agent and the offensive cybersecurity agent, we can still find a better workflow via iterative workflow refinement. We evaluate each workflow 5 times and report the best average pass@$1$ score as the performance.