SCRUBD: Smart Contracts Reentrancy and Unhandled Exceptions Vulnerability Dataset

TL;DR

SCRUBD addresses the need for a standardized benchmark for Ethereum smart-contract vulnerabilities by combining real-world crowdsourced labels (SCRUBD/CD) with meticulously crafted synthetic RE cases (SCRUBD/SD). It evaluates six vulnerability-detection tools, revealing that Slither and Sailfish provide strongest performance in different data regimes and that existing datasets contain labeling issues that SCRUBD helps to clarify. The dataset enables unbiased tool comparison, supports machine-learning based vulnerability detection, and offers a comprehensive suite to stress-test RE and UX coverage. By making SCRUBD publicly available, the work advances reproducible security benchmarking in the smart-contract domain and outlines plans for further expansion and automation.

Abstract

Smart Contracts (SCs) handle transactions in the Ethereum blockchain worth millions of United States dollars, making them a lucrative target for attackers seeking to exploit vulnerabilities and steal funds. The Ethereum community has developed a rich set of tools to detect vulnerabilities in SCs, including reentrancy (RE) and unhandled exceptions (UX). A dataset of SCs labelled with vulnerabilities is needed to evaluate the tools' efficacy. Existing SC datasets with labelled vulnerabilities have limitations, such as covering only a limited range of vulnerability scenarios and containing incorrect labels. As a result, there is a lack of a standardized dataset to compare the performances of these tools. SCRUBD aims to fill this gap. We present a dataset of real-world SCs and synthesized SCs labelled with RE and UX. The real-world SC dataset is labelled through crowdsourcing, followed by manual inspection by an expert, and covers both RE and UX vulnerabilities. On the other hand, the synthesized dataset is carefully crafted to cover various RE scenarios only. Using SCRUBD we compared the performance of six popular vulnerability detection tools. Based on our study, we found that Slither outperforms other tools on a crowdsourced dataset in detecting RE vulnerabilities, while Sailfish outperforms other tools on a manually synthesized dataset for detecting RE. For UX vulnerabilities, Slither outperforms all other tools.

Figures (2)

• Figure 1: Methodology to create SCRUBD/CD (Conkas: CK, Mythril: MY, Sailfish: SF, Slither: SL, Solhint: SH)
• Figure 2: F1-Scores of Existing Tools using SCRUBD (Slither: SL, Sailfish: SF, Solhint: SH, Mythril: MY, Conkas: CK, Smartcheck: SC) — Higher the Better