A Comparative Evaluation of Automated Analysis Tools for Solidity Smart Contracts
Zhiyuan Wei, Xianhao Zhang, Jing Sun, Zijian Zhang, Liehuang Zhu
TL;DR
The paper tackles the challenge of evaluating automated solidity smart-contract analysis tools by constructing a public benchmark and deploying a rigorous ISO/IEC 25010-based quality framework enhanced with entropy weighting (EWM) and analytic hierarchy process (AHP) to produce objective tool scores. It assembles two datasets—a 389-labelled contract set and a 20,000 real-world contract sample—to assess 13 tools across formal verification, symbolic execution, fuzzing, IR, and machine learning methods. The study identifies Mythril, Slither, and Solhint as top performers across multiple metrics, while also highlighting the benefits and limitations of combining tools for comprehensive vulnerability detection. The findings offer practical guidance for practitioners on tool selection and reveal trends in vulnerability prevalence over time, supporting more secure and reliable smart-contract development. The work also provides public benchmarks and a framework adaptable to future tools and vulnerability categories, encouraging broader cross-language analysis and ongoing security improvements.
Abstract
Blockchain smart contracts have emerged as a transformative force in the digital realm, spawning a diverse range of compelling applications. Since solidity smart contracts across various domains manage trillions of dollars in virtual coins, they become a prime target for attacks. One of the primary challenges is keeping abreast of the latest techniques and tools for developing secure smart contracts and examining those already deployed. In this paper, we seek to address these challenges from four aspects: (1) We begin by examining ten automatic tools, specifically focusing on their methodologies and their ability to identify vulnerabilities in solidity smart contracts. (2) We propose a novel criterion for evaluating these tools, based on the ISO/IEC 25010 standard. (3) To facilitate the evaluation of the selected tools, we construct a benchmark that encompasses two distinct datasets: a collection of 389 labelled smart contracts and a scaled set of 20,000 unique cases from real-world contracts. (4) We provide a comparison of the selected tools, offering insights into their strengths and weaknesses and highlighting areas where further improvements are needed. Through this evaluation, we hope to provide developers and researchers with valuable guidance on selecting and using smart contract analysis tools and contribute to the ongoing efforts to improve the security and reliability of smart contracts.