Do Automated Fixes Truly Mitigate Smart Contract Exploits?

TL;DR

The paper tackles the challenge of evaluating automated program repair for smart contracts by introducing a ground-truth exploit-based framework. It systematically assesses 7 APR tools on a common dataset of $143$ vulnerable contracts, with $91$ manually crafted exploits, measuring exploit mitigation rates and patch quality. Results reveal substantial variability across tools (from $29\%$ to $74\%$ mitigation) and highlight trade-offs between patch functionality preservation and security, particularly between bytecode- and source-level repair. The work emphasizes the need for standardized vulnerability representations, robust patch validation, and ensemble repair strategies to advance practical, secure smart contracts. The publicly released datasets and replication package underpin reproducibility and future research in secure automated repair.

Abstract

Automated Program Repair (APR) for smart contract security promises to automatically mitigate smart contract vulnerabilities responsible for billions in financial losses. However, the true effectiveness of this research in addressing smart contract exploits remains uncharted territory. This paper bridges this critical gap by introducing a novel and systematic experimental framework for evaluating exploit mitigation of program repair tools for smart contracts. We qualitatively and quantitatively analyze 20 state-of-the-art APR tools using a dataset of 143 vulnerable smart contracts, for which we manually craft 91 executable exploits. We are the very first to define and measure the essential "exploit mitigation rate" , giving researchers and practitioners a real sense of effectiveness of cutting edge techniques. Our findings reveal substantial disparities in the state of the art, with an exploit mitigation rate ranging from a low of 29% to a high of 74%. Our study identifies systemic limitations, such as inconsistent functionality preservation, that must be addressed in future research on program repair for smart contracts.

Figures (3)

• Figure 1: Overview of the general workflow of an automated program repair tool for smart contracts.
• Figure 2: RQ3: Our novel methodology for evaluation via actual, executable exploits, a core contribution to the field of smart contract repair.
• Figure 3: Detection Rate and Exploit Mitigation Rate for each APR tool. Detection Rate represents the percentage of vulnerabilities with exploits (91) identified by each tool. Mitigation Rate reflects the percentage of exploits successfully mitigated out of the same total. Results are specific to vulnerabilities with exploits and, therefore, differ from Table \ref{['tab:over_comparison_rq2']}.