Are We There Yet? Unraveling the State-of-the-Art Smart Contract Fuzzers
Shuohan Wu, Zihao Li, Luyi Yan, Weimin Chen, Muhui Jiang, Chenxu Wang, Xiapu Luo, Hao Zhou
TL;DR
This work addresses the lack of systematic evaluation for smart contract fuzzers by combining a comprehensive literature review with an empirical study. It introduces a unified 2,000-contract benchmark and ground-truth labeling to fairly compare 11 state-of-the-art fuzzers across throughput, vulnerability detection, code coverage, and overhead. The findings reveal that current fuzzers struggle with accurate vulnerability detection (high false positives/negatives) and that throughput and seed quality are pivotal for effective exploration, with ConFuzzius often leading in several metrics. The study also collects industrial perspectives through audits and surveys, highlighting a demand for customizable test oracles and online fuzzing, and it releases the benchmark and code to foster future development. Overall, the paper provides actionable guidance for designing more effective fuzzers and for aligning academic research with industry needs.
Abstract
Given the growing importance of smart contracts in various applications, ensuring their security and reliability is critical. Fuzzing, an effective vulnerability detection technique, has recently been widely applied to smart contracts. Despite numerous studies, a systematic investigation of smart contract fuzzing techniques remains lacking. In this paper, we fill this gap by: 1) providing a comprehensive review of current research in contract fuzzing, and 2) conducting an in-depth empirical study to evaluate state-of-the-art contract fuzzers' usability. To guarantee a fair evaluation, we employ a carefully-labeled benchmark and introduce a set of pragmatic performance metrics, evaluating fuzzers from five complementary perspectives. Based on our findings, we provide direction for the future research and development of contract fuzzers.