Mapping AI Benchmark Data to Quantitative Risk Estimates Through Expert Elicitation

TL;DR

This paper tackles the gap between AI benchmark capabilities and real-world harms by proposing a risk-modeling approach that maps Cybench benchmark performance to probabilistic estimates within a cyber-risk scenario. It employs the IDEA protocol for structured expert elicitation, linking a malware-development step to tasks of increasing difficulty (First Solve Time) and aggregating expert judgments via Bayesian interpolation. The pilot reveals substantial inter-expert divergence and suggests that current LLMs provide modest uplift (roughly $5$–$10 ext{%}$) in malware success probability, with larger uplift ($40$–$65 ext{%}$) if an LLM can master the hardest benchmark tasks, albeit with wide uncertainty. The work highlights the need for more explicit risk definitions, uplift-focused evaluations, and risk-informed benchmarks to better connect model capabilities with potential real-world harms.

Abstract

The literature and multiple experts point to many potential risks from large language models (LLMs), but there are still very few direct measurements of the actual harms posed. AI risk assessment has so far focused on measuring the models' capabilities, but the capabilities of models are only indicators of risk, not measures of risk. Better modeling and quantification of AI risk scenarios can help bridge this disconnect and link the capabilities of LLMs to tangible real-world harm. This paper makes an early contribution to this field by demonstrating how existing AI benchmarks can be used to facilitate the creation of risk estimates. We describe the results of a pilot study in which experts use information from Cybench, an AI benchmark, to generate probability estimates. We show that the methodology seems promising for this purpose, while noting improvements that can be made to further strengthen its application in quantitative AI risk assessment.

Figures (3)

• Figure 2: | Components of the risk scenario | The risk scenario has six steps, starting with the existence of an actor and their attempts at executing the risk scenario, and ending with the economic damage ensuing from the successful completion of the attack. Between those two is a set of probabilities for each step, conditional on completing the prior steps.
• Figure 3: | Mean probability estimates over increasing FST | Relationship between FST (First Solve Time) of the hardest task an LLM can solve in Cybench and the estimated probability of a cybercrime group successfully developing and deploying malware with that LLM's assistance. The baseline probability without LLM assistance is 25%. Reference points show the highest FST that current models (o1, Claude 3.5 Sonnet, and GPT-4o) can consistently solve.
• Figure 4: | Mean probability estimates over increasing FST within individual groups | Comparison of mean probability estimates between Groups A and B, showing how each group's interpretation of LLM capabilities led to different assessments. Group A estimated higher probabilities, viewing LLM capability at solving complex CTF tasks as a significant advantage for cybercrime groups. Group B estimated lower probabilities, considering CTF performance as only minimally indicative of real-world malware development capabilities. Confidence intervals are only shown for the combined group (A+B) as individual groups had insufficient data points for meaningful CI calculation.