Do you still need a manual smart contract audit?

TL;DR

The paper investigates whether large language models can meaningfully augment security audits of DeFi smart contracts. By evaluating GPT-4-32k and Claude-v1.3-100k on 52 compromised contracts across 38 vulnerability types, the study shows a reachable but imperfect $40\%$ hit rate with substantial false positives, underscoring that human auditors remain essential. Through mutation testing with five synthetic secure contracts, the authors demonstrate that carefully designed prompts and chain-of-thought strategies can yield higher true positives (up to $78.7\%$ in some configurations) and reveal how model temperature and context length influence performance. The work highlights practical prompt-engineering approaches that can streamline audits while acknowledging limitations in coverage and reliability, thereby positioning LLMs as scalable assistants rather than replacements for skilled auditors. Overall, the study contributes to the evolution of smart contract security workflows by enabling rapid, broad-spectrum vulnerability screening and guiding future improvements in robustness and evaluation across diverse datasets.

Abstract

We investigate the feasibility of employing large language models (LLMs) for conducting the security audit of smart contracts, a traditionally time-consuming and costly process. Our research focuses on the optimization of prompt engineering for enhanced security analysis, and we evaluate the performance and accuracy of LLMs using a benchmark dataset comprising 52 Decentralized Finance (DeFi) smart contracts that have previously been compromised. Our findings reveal that, when applied to vulnerable contracts, both GPT-4 and Claude models correctly identify the vulnerability type in 40% of the cases. However, these models also demonstrate a high false positive rate, necessitating continued involvement from manual auditors. The LLMs tested outperform a random model by 20% in terms of F1-score. To ensure the integrity of our study, we conduct mutation testing on five newly developed and ostensibly secure smart contracts, into which we manually insert two and 15 vulnerabilities each. This testing yielded a remarkable best-case 78.7% true positive rate for the GPT-4-32k model. We tested both, asking the models to perform a binary classification on whether a contract is vulnerable, and a non-binary prompt. We also examined the influence of model temperature variations and context length on the LLM's performance. Despite the potential for many further enhancements, this work lays the groundwork for a more efficient and economical approach to smart contract security audits.

Figures (6)

• Figure 1: Distribution of the token lengths of the 52 attacked smart contracts. Most of the vulnerable smart contracts fit into the context length of GPT-4-32k.
• Figure 2: Mutation testing with LLM temperature set to 0. GPT's performance decreases detecting 13/15 vulnerabilities correctly but with an increase in false positives. Claude's performance is negatively affected, with an F1 score inferior to the random model.
• Figure 3: Mutation testing results at an LLM temperature of 1. A noticeable increase in the F1 scores for both models is observed. Remarkably, Claude's F1 score saw a fourfold increase, surging from 0.125 at a temperature of 0 to 0.500 at a temperature of 1.
• Figure 4: Performance comparison of the three models in long context runs on the AirdropFaucet contract. GPT-4 and Claude show a higher number of false negatives, while Claude outperforms GPT-4 in terms of true positives and F1 score, suggesting better effectiveness in detecting vulnerabilities for longer contexts.
• Figure 5: Case Study 1: Level Finance Hack --- Analysis of the Absence of Code Logic Vulnerability in the claimMultiple Function that enabled double-spending and resulted in a loss of 3,345 BNB, alongside GPT's proposed solution for updating the claimed state variable.