Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Reentrancy Detection in the Age of LLMs

Dalila Ressi, Alvise Spanò, Matteo Rizzo, Lorenzo Benetollo, Sabina Rossi

Abstract

Reentrancy remains one of the most critical classes of vulnerabilities in Ethereum smart contracts, yet widely used detection tools and datasets continue to reflect outdated patterns and obsolete Solidity versions. This paper adopts a dependability-oriented perspective on reentrancy detection in Solidity 0.8+, assessing how reliably state-of-the-art static analyzers and AI-based techniques operate on modern code by putting them to the test on two fronts. We construct two manually verified benchmarks: an Aggregated Benchmark of 432 real-world contracts, consolidated and relabeled from prior datasets, and a Reentrancy Scenarios Dataset (RSD) of \chadded{143} handcrafted minimal working examples designed to isolate and stress-test individual reentrancy patterns. We then evaluate 12 formal-methods-based tools, 10 machine-learning models, and 9 large language models. On the Aggregated Benchmark, traditional tools and ML models achieve up to 0.87 F1, while the best LLMs reach 0.96 in a zero-shot setting. On the RSD, most tools fail on multiple scenarios, the top performer achieving an F1 of 0.76, whereas the strongest model attains 0.82. Overall, our results indicate that leading LLMs outperform the majority of existing detectors, highlighting concerning gaps in the robustness and maintainability of current reentrancy-analysis tools.

Reentrancy Detection in the Age of LLMs

Abstract

Reentrancy remains one of the most critical classes of vulnerabilities in Ethereum smart contracts, yet widely used detection tools and datasets continue to reflect outdated patterns and obsolete Solidity versions. This paper adopts a dependability-oriented perspective on reentrancy detection in Solidity 0.8+, assessing how reliably state-of-the-art static analyzers and AI-based techniques operate on modern code by putting them to the test on two fronts. We construct two manually verified benchmarks: an Aggregated Benchmark of 432 real-world contracts, consolidated and relabeled from prior datasets, and a Reentrancy Scenarios Dataset (RSD) of \chadded{143} handcrafted minimal working examples designed to isolate and stress-test individual reentrancy patterns. We then evaluate 12 formal-methods-based tools, 10 machine-learning models, and 9 large language models. On the Aggregated Benchmark, traditional tools and ML models achieve up to 0.87 F1, while the best LLMs reach 0.96 in a zero-shot setting. On the RSD, most tools fail on multiple scenarios, the top performer achieving an F1 of 0.76, whereas the strongest model attains 0.82. Overall, our results indicate that leading LLMs outperform the majority of existing detectors, highlighting concerning gaps in the robustness and maintainability of current reentrancy-analysis tools.

Paper Structure

This paper contains 25 sections, 5 equations, 1 figure, 4 tables.

Table of Contents

  1. Introduction
  2. Contributions
  3. Threat Model
  4. Attacker Capabilities and Goals
  5. Operational Definition of Reentrancy
  6. Motivation
  7. Why manually labeled datasets were our starting point
  8. Why we extracted the rules of existing tools
  9. Why this fragmentation matters
  10. Implications for ground truth
  11. Dataset Construction
  12. Aggregated Benchmark
  13. Reentrancy Scenarios Dataset (RSD)
  14. Labeling Procedure
  15. Methodology
  16. ...and 10 more sections

Figures (1)

  • Figure 1: Comparison between upset plots for Aggregated Benchmark and RSD datasets. This type of plot serves as an alternative to Venn diagrams for visualizing intersections and overlaps among more than three sets. The histograms indicate the number of samples per row and column, while the dots show which tools flagged the contracts within each set.