Automated Vulnerability Injection in Solidity Smart Contracts: A Mutation-Based Approach for Benchmark Development

TL;DR

This work tackles the problem of evaluating vulnerability detectors for Solidity smart contracts by generating realistic, large-scale benchmarks through a mutation-based approach. The authors introduce MuSe, which extends SuMo with six pattern-driven mutation operators to inject six known vulnerabilities into real-world contracts, then assess detection using Slither on a large dataset (smartbugs-wild). Key findings show heterogeneous injection success across operators, substantial benchmark expansion, and significant gaps in static analysis, especially for complex patterns, underscoring the need for more robust detectors. The work provides an open-source replication package and demonstrates that mutation-based benchmark generation can meaningfully improve evaluation of vulnerability detectors and guide future security tooling improvements.

Abstract

The security of smart contracts is critical in blockchain systems, where even minor vulnerabilities can lead to substantial financial losses. Researchers proposed several vulnerability detection tools evaluated using existing benchmarks. However, most benchmarks are outdated and focus on a narrow set of vulnerabilities. This work evaluates whether mutation seeding can effectively inject vulnerabilities into Solidity-based smart contracts and whether state-of-the-art static analysis tools can detect the injected flaws. We aim to automatically inject vulnerabilities into smart contracts to generate large and wide benchmarks. We propose MuSe, a tool to generate vulnerable smart contracts by leveraging pattern-based mutation operators to inject six vulnerability types into real-world smart contracts. We analyzed these vulnerable smart contracts using Slither, a static analysis tool, to determine its capacity to identify them and assess their validity. The results show that each vulnerability has a different injection rate. Not all smart contracts can exhibit some vulnerabilities because they lack the prerequisites for injection. Furthermore, static analysis tools fail to detect all vulnerabilities injected using pattern-based mutations, underscoring the need for enhancements in static analyzers and demonstrating that benchmarks generated by mutation seeding tools can improve the evaluation of detection tools.

Figures (5)

• Figure 1: Summary of the Research Method.
• Figure 2: Incorrect modifier to restrict owner's access.
• Figure 3: Clause with two conditions combined using OR.
• Figure 4: Two examples of unused return.
• Figure 5: Delegatecall to untrusted callee.