Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Prompt to Pwn: Automated Exploit Generation for Smart Contracts

Zeke Xiao, Qin Wang, Yuekang Li, Shiping Chen

TL;DR

This work addresses the difficulty of patching immutable smart contracts by introducing ReX, an end-to-end framework that couples LLM-based exploit synthesis with the Foundry toolchain to automatically generate, compile, run, and verify PoCs. Through two datasets, SmartBugs-Curated and Web3-AEG, and evaluation across five LLMs, the study reveals strong performance on single-contract exploits but limited generalization to cross-contract scenarios. It identifies model capability as the primary driver of exploit success, with vulnerability type and prompt design playing secondary roles, and it exposes gaps in current defenses against LLM-driven exploit generation. The paper contributes a public PoC dataset, an automated AEG pipeline, and defense recommendations, highlighting practical implications for securing smart contracts in real-world ecosystems.

Abstract

Smart contracts are important for digital finance, yet they are hard to patch once deployed. Prior work mostly studies LLMs for vulnerability detection, leaving their automated exploit generation (AEG) capability unclear. This paper closes that gap with \textsc{ReX}, a framework that links LLM-based exploit synthesis to the Foundry stack for end-to-end generation, compilation, execution, and verification. Five recent LLMs are evaluated across eight common vulnerability classes, supported by a curated dataset of 38{+} real incident PoCs and three automation aids: prompt refactoring, a compiler feedback loop, and templated test harnesses. Results indicate strong performance on single-contract PoCs and weak performance on cross-contract attacks; outcomes depend mainly on the model and bug type, with code structure and prompt tuning contributing little. The study also surfaces gaps in current defenses against LLM-driven AEG, pointing to the need for stronger protections.

Prompt to Pwn: Automated Exploit Generation for Smart Contracts

TL;DR

This work addresses the difficulty of patching immutable smart contracts by introducing ReX, an end-to-end framework that couples LLM-based exploit synthesis with the Foundry toolchain to automatically generate, compile, run, and verify PoCs. Through two datasets, SmartBugs-Curated and Web3-AEG, and evaluation across five LLMs, the study reveals strong performance on single-contract exploits but limited generalization to cross-contract scenarios. It identifies model capability as the primary driver of exploit success, with vulnerability type and prompt design playing secondary roles, and it exposes gaps in current defenses against LLM-driven exploit generation. The paper contributes a public PoC dataset, an automated AEG pipeline, and defense recommendations, highlighting practical implications for securing smart contracts in real-world ecosystems.

Abstract

Smart contracts are important for digital finance, yet they are hard to patch once deployed. Prior work mostly studies LLMs for vulnerability detection, leaving their automated exploit generation (AEG) capability unclear. This paper closes that gap with \textsc{ReX}, a framework that links LLM-based exploit synthesis to the Foundry stack for end-to-end generation, compilation, execution, and verification. Five recent LLMs are evaluated across eight common vulnerability classes, supported by a curated dataset of 38{+} real incident PoCs and three automation aids: prompt refactoring, a compiler feedback loop, and templated test harnesses. Results indicate strong performance on single-contract PoCs and weak performance on cross-contract attacks; outcomes depend mainly on the model and bug type, with code structure and prompt tuning contributing little. The study also surfaces gaps in current defenses against LLM-driven AEG, pointing to the need for stronger protections.

Paper Structure

This paper contains 32 sections, 4 figures, 13 tables.

Table of Contents

  1. Introduction
  2. Background
  3. Approach
  4. Design of ReX
  5. Datasets for Evaluation
  6. Threat Model
  7. Safety Invariants for Automated PoC Validation
  8. Evaluation
  9. Exp1: Benchmarking on SmartBugs-Curated.
  10. Exp2: Real-world exploits with Web3-AEG.
  11. Answer to RQ1: can LLMs perform AEGs?
  12. Key Factors Affecting LLM Performance
  13. Factor 1: LLM capabilities and failure patterns
  14. Factor 2: properties of target contracts
  15. Factor 3: Vulnerability Type
  16. ...and 17 more sections

Figures (4)

  • Figure 1: End-to-end workflow of ReX. The system iteratively synthesizes, repairs, and validates exploit PoCs until Foundry confirms a working attack.
  • Figure 2: AEG Success Rate by Vulnerability Type
  • Figure 3: Metrics for Contracts with (upper) and without (bottom) AEG
  • Figure 4: Contract metrics of WEB3-AEG

Theorems & Definitions (1)

  • definition 1: AEG success