Smart Contract and DeFi Security Tools: Do They Meet the Needs of Practitioners?

TL;DR

The paper evaluates whether automated smart contract security tools meet DeFi practitioners' needs by combining a real-world attack-based empirical study of five state-of-the-art tools with a practitioner survey. It finds that automated tools could have prevented only about 8% of 127 high-impact exploits, with all preventable cases tied to reentrancy, while major gaps remain for logic-related and protocol-layer vulnerabilities. The work highlights usability and coverage issues, shows different tool preferences for developers and auditors, and argues for semi-automated tools and specialized tooling tailored to practical workflows. These findings underscore the need for broader vulnerability coverage and realism-driven benchmarks to meaningfully improve security in DeFi ecosystems.

Abstract

The growth of the decentralized finance (DeFi) ecosystem built on blockchain technology and smart contracts has led to an increased demand for secure and reliable smart contract development. However, attacks targeting smart contracts are increasing, causing an estimated \$6.45 billion in financial losses. Researchers have proposed various automated security tools to detect vulnerabilities, but their real-world impact remains uncertain. In this paper, we aim to shed light on the effectiveness of automated security tools in identifying vulnerabilities that can lead to high-profile attacks, and their overall usage within the industry. Our comprehensive study encompasses an evaluation of five SoTA automated security tools, an analysis of 127 high-impact real-world attacks resulting in \$2.3 billion in losses, and a survey of 49 developers and auditors working in leading DeFi protocols. Our findings reveal a stark reality: the tools could have prevented a mere 8% of the attacks in our dataset, amounting to \$149 million out of the \$2.3 billion in losses. Notably, all preventable attacks were related to reentrancy vulnerabilities. Furthermore, practitioners distinguish logic-related bugs and protocol layer vulnerabilities as significant threats that are not adequately addressed by existing security tools. Our results emphasize the need to develop specialized tools catering to the distinct demands and expectations of developers and auditors. Further, our study highlights the necessity for continuous advancements in security tools to effectively tackle the ever-evolving challenges confronting the DeFi ecosystem.

Figures (13)

• Figure 1: Point-to-point comparison of related work on evaluating automated security tools. *: This paper categorizes bugs in machine unauditable bugs. Two tools, Slither and Oyente, were used not to measure false positives and false negatives, but to validate whether the tools were able to detect MUB bugs as defined in the study.
• Figure 2: Summary of vulnerability categories and the number of corresponding exploits in the Zhou et al. dataset defisok. ● indicates tool support for a corresponding vulnerability type. An empty cell indicates that a tool does not support the respective vulnerability. SC: Smart Contract Layer, PRO: Protocol Layer. We exclude vulnerability types that (1) the tools cannot support and (2) do not exist in the dataset. Note that one exploit can be caused due to multiple vulnerabilities.
• Figure 3: Overall descriptive statistics of the analysed attacks.
• Figure 4: Evaluating the effectiveness of security tools.
• Figure 5: Survey participant demographics.