Combining Fine-Tuning and LLM-based Agents for Intuitive Smart Contract Auditing with Justifications

TL;DR

iAudit addresses the challenge of accurately auditing smart contracts by merging domain-focused two-stage fine-tuning with LLM-based agents for reasoning. Detector performs intuitive vulnerability judgments via multi-prompt training and majority voting, while Reasoner generates multiple explanations which Ranker and Critic iteratively refine to select the best root cause. On a real-world dataset, iAudit achieves a high F1 of 0.9121 and accuracy of 0.9111, with explanation consistency of 37.99%, outperforming zero-shot LLMs and traditional fine-tuned baselines. The work highlights the value of structured prompting, two-stage training, and agent-based reasoning, while acknowledging limitations such as potential hallucinations and mixed benefits from additional contextual information like call graphs. Overall, iAudit demonstrates a promising path toward intuitive, interpretable smart contract auditing with robust performance and explainability gains.

Abstract

Smart contracts are decentralized applications built atop blockchains like Ethereum. Recent research has shown that large language models (LLMs) have potential in auditing smart contracts, but the state-of-the-art indicates that even GPT-4 can achieve only 30% precision (when both decision and justification are correct). This is likely because off-the-shelf LLMs were primarily pre-trained on a general text/code corpus and not fine-tuned on the specific domain of Solidity smart contract auditing. In this paper, we propose iAudit, a general framework that combines fine-tuning and LLM-based agents for intuitive smart contract auditing with justifications. Specifically, iAudit is inspired by the observation that expert human auditors first perceive what could be wrong and then perform a detailed analysis of the code to identify the cause. As such, iAudit employs a two-stage fine-tuning approach: it first tunes a Detector model to make decisions and then tunes a Reasoner model to generate causes of vulnerabilities. However, fine-tuning alone faces challenges in accurately identifying the optimal cause of a vulnerability. Therefore, we introduce two LLM-based agents, the Ranker and Critic, to iteratively select and debate the most suitable cause of vulnerability based on the output of the fine-tuned Reasoner model. To evaluate iAudit, we collected a balanced dataset with 1,734 positive and 1,810 negative samples to fine-tune iAudit. We then compared it with traditional fine-tuned models (CodeBERT, GraphCodeBERT, CodeT5, and UnixCoder) as well as prompt learning-based LLMs (GPT4, GPT-3.5, and CodeLlama-13b/34b). On a dataset of 263 real smart contract vulnerabilities, iAudit achieves an F1 score of 91.21% and an accuracy of 91.11%. The causes generated by iAudit achieved a consistency of about 38% compared to the ground truth causes.

Figures (9)

• Figure 1: An overview of iAudit, featuring its four roles: Detector, Reasoner, Ranker, and Critic.
• Figure 2: The Prompt Template Used by Detector.
• Figure 3: The Prompt Template Used by Reasoner.
• Figure 4: Detailed Multiple Prompts for Detector and Reasoner.
• Figure 5: Data Enhancement for Expanding Vulnerability Explanations based on GPT-3.5.