Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

AuditGPT: Auditing Smart Contracts with ChatGPT

Shihao Xia, Shuai Shao, Mengting He, Tingting Yu, Linhai Song, Yiying Zhang

TL;DR

AuditGPT addresses the challenge of verifying ERC compliance in Ethereum smart contracts by leveraging large language models to automate rule auditing. Built on an empirical study of 222 ERC rules across four popular standards, the system uses a divide-and-conquer workflow with per-function code slicing, rule extraction into YAML, and specialized prompts to assess compliance. Evaluation on large and ground-truth datasets shows AuditGPT detects large numbers of violations with high accuracy, outperforming automated baselines and a manual expert service while reducing cost and time by orders of magnitude. The work demonstrates that structured prompt engineering and modular analysis enable LLMs to tackle semantically rich, contract-specific verification tasks, offering scalable improvements over current practices.

Abstract

To govern smart contracts running on Ethereum, multiple Ethereum Request for Comment (ERC) standards have been developed, each containing a set of rules to guide the behaviors of smart contracts. Violating the ERC rules could cause serious security issues and financial loss, signifying the importance of verifying smart contracts follow ERCs. Today's practices of such verification are to either manually audit each single contract or use expert-developed, limited-scope program-analysis tools, both of which are far from being effective in identifying ERC rule violations. This paper presents a tool named AuditGPT that leverages large language models (LLMs) to automatically and comprehensively verify ERC rules against smart contracts. To build AuditGPT, we first conduct an empirical study on 222 ERC rules specified in four popular ERCs to understand their content, their security impacts, their specification in natural language, and their implementation in Solidity. Guided by the study, we construct AuditGPT by separating the large, complex auditing process into small, manageable tasks and design prompts specialized for each ERC rule type to enhance LLMs' auditing performance. In the evaluation, AuditGPT successfully pinpoints 418 ERC rule violations and only reports 18 false positives, showcasing its effectiveness and accuracy. Moreover, AuditGPT beats an auditing service provided by security experts in effectiveness, accuracy, and cost, demonstrating its advancement over state-of-the-art smart-contract auditing practices.

AuditGPT: Auditing Smart Contracts with ChatGPT

TL;DR

AuditGPT addresses the challenge of verifying ERC compliance in Ethereum smart contracts by leveraging large language models to automate rule auditing. Built on an empirical study of 222 ERC rules across four popular standards, the system uses a divide-and-conquer workflow with per-function code slicing, rule extraction into YAML, and specialized prompts to assess compliance. Evaluation on large and ground-truth datasets shows AuditGPT detects large numbers of violations with high accuracy, outperforming automated baselines and a manual expert service while reducing cost and time by orders of magnitude. The work demonstrates that structured prompt engineering and modular analysis enable LLMs to tackle semantically rich, contract-specific verification tasks, offering scalable improvements over current practices.

Abstract

To govern smart contracts running on Ethereum, multiple Ethereum Request for Comment (ERC) standards have been developed, each containing a set of rules to guide the behaviors of smart contracts. Violating the ERC rules could cause serious security issues and financial loss, signifying the importance of verifying smart contracts follow ERCs. Today's practices of such verification are to either manually audit each single contract or use expert-developed, limited-scope program-analysis tools, both of which are far from being effective in identifying ERC rule violations. This paper presents a tool named AuditGPT that leverages large language models (LLMs) to automatically and comprehensively verify ERC rules against smart contracts. To build AuditGPT, we first conduct an empirical study on 222 ERC rules specified in four popular ERCs to understand their content, their security impacts, their specification in natural language, and their implementation in Solidity. Guided by the study, we construct AuditGPT by separating the large, complex auditing process into small, manageable tasks and design prompts specialized for each ERC rule type to enhance LLMs' auditing performance. In the evaluation, AuditGPT successfully pinpoints 418 ERC rule violations and only reports 18 false positives, showcasing its effectiveness and accuracy. Moreover, AuditGPT beats an auditing service provided by security experts in effectiveness, accuracy, and cost, demonstrating its advancement over state-of-the-art smart-contract auditing practices.
Paper Structure (21 sections, 3 figures, 4 tables)

This paper contains 21 sections, 3 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Background
  3. Ethereum and Solidity Smart Contracts
  4. Ethereum Request for Comment (ERC)
  5. Related work
  6. Empirical Study on ERC rules
  7. Methodology
  8. Rule Content (What)
  9. Violation Impact (Why)
  10. Specification and Implementation (How)
  11. AuditGPT Design
  12. Overview
  13. ERC Rule Extraction
  14. Code Slicing
  15. Optimization Mechanisms
  16. ...and 6 more sections

Figures (3)

  • Figure 1: An ERC20 rule violation that can be exploited to steal tokens. (Code simplified for illustration purpose.)
  • Figure 2: An ERC1155 rule violation with a high-security impact. (Code simplified for illustration purpose.)
  • Figure 3: Evaluation results on the ground-truth dataset with each design point deactivated. (W.O.: without, RE: rule sequentialization, S: specialization, OS: one shot, CM: breaking down, CS: code slicing)