Frontier AI's Impact on the Cybersecurity Landscape

TL;DR

This work confronts the question of whether frontier AI will benefit attackers or defenders in cybersecurity by applying a marginal risk framework across quantitative benchmarks, qualitative literature, empirical agent evaluations, and expert surveys. It finds that AI currently yields greater impact for attacks than defenses, with empirical evidence showing defenders lagging in large-scale, domain-specific security tasks. The paper offers concrete calls to action—develop richer, real-world benchmarks; build defense-focused AI agents with secure-by-design guarantees; enhance pre-deployment testing and transparency; and improve human-centered education and defenses. It argues that while attackers may gain an early advantage, strategic defenses leveraging AI, formal verification, and robust benchmarks can shift the balance toward more secure systems over time.

Abstract

The impact of frontier AI (i.e., AI agents and foundation models) in cybersecurity is rapidly increasing. In this paper, we comprehensively analyze this trend through multiple aspects: quantitative benchmarks, qualitative literature review, empirical evaluation, and expert survey. Our analyses consistently show that AI's capabilities and applications in attacks have exceeded those on the defensive side. Our empirical evaluation of widely used agent systems on cybersecurity benchmarks highlights that current AI agents struggle with flexible workflow planning and using domain-specific tools for complex security analysis -- capabilities particularly critical for defensive applications. Our expert survey of AI and security researchers and practitioners indicates a prevailing view that AI will continue to benefit attackers over defenders, though the gap is expected to narrow over time. These results show the urgent need to evaluate and mitigate frontier AI's risks, steering it towards benefiting cyber defenses. Responding to this need, we provide concrete calls to action regarding: the construction of new cybersecurity benchmarks, the development of AI agents for defense, the design of provably secure AI agents, the improvement of pre-deployment security testing and transparency, and the strengthening of user-oriented education and defenses. Our paper summary and blog are available at https://rdi.berkeley.edu/frontier-ai-impact-on-cybersecurity/.

Figures (7)

• Figure 1: Software engineering performance evolvement of frontier AI over time.
• Figure 2: A comprehensive categorization of attacks (red canvas) and defenses (blue canvas) with frontier AI's qualitative impact. AI has a larger impact on final attack goals than on specific stages. The disparity exists because most reports disclose only final impacts rather than attack procedures. For precision, we include only reports with clear specifications for individual attack steps.
• Figure 3: Success Rate of each attack step for real-world tasks.
• Figure 4: Top 10 commands executed by OpenHands-Pen.
• Figure 5: Top 10 commands executed by OpenHands-Def.