LLM Cyber Evaluations Don't Capture Real-World Risk

TL;DR

<3-5 sentence high-level summary>This work argues that assessing the real-world risks of LLM-based cybersecurity capabilities requires integrating threat-actor adoption dynamics and impact analysis, not just measuring nominal capabilities. It introduces a three-part risk framework—hazard identification, frequency analysis, and consequence analysis—and demonstrates its application through a case study on LLMs as cybersecurity assistants, revealing that high compliance with prompts does not necessarily translate into high real-world risk. The findings show moderate risk for the studied use case and emphasize the need for academia–industry collaboration, modeling realistic attacker behavior, and incorporating economic metrics into evaluations. By linking capability evaluation to real-world impact, the paper offers a path toward more effective mitigation of LLM-enabled cybersecurity risks.

Abstract

Large language models (LLMs) are demonstrating increasing prowess in cybersecurity applications, creating creating inherent risks alongside their potential for strengthening defenses. In this position paper, we argue that current efforts to evaluate risks posed by these capabilities are misaligned with the goal of understanding real-world impact. Evaluating LLM cybersecurity risk requires more than just measuring model capabilities -- it demands a comprehensive risk assessment that incorporates analysis of threat actor adoption behavior and potential for impact. We propose a risk assessment framework for LLM cyber capabilities and apply it to a case study of language models used as cybersecurity assistants. Our evaluation of frontier models reveals high compliance rates but moderate accuracy on realistic cyber assistance tasks. However, our framework suggests that this particular use case presents only moderate risk due to limited operational advantages and impact potential. Based on these findings, we recommend several improvements to align research priorities with real-world impact assessment, including closer academia-industry collaboration, more realistic modeling of attacker behavior, and inclusion of economic metrics in evaluations. This work represents an important step toward more effective assessment and mitigation of LLM-enabled cybersecurity risks.