ParaVul: A Parallel Large Language Model and Retrieval-Augmented Framework for Smart Contract Vulnerability Detection

TL;DR

ParaVul addresses the challenge of accurate and scalable smart contract vulnerability detection by combining a sparsified, low-rank fine-tuned LLM (SLoRA) with a hybrid retrieval-augmented system (dense + BM25) and a meta-learning gated verifier. The framework runs LLM and RAG in parallel, fusing their outputs to improve robustness and reduce false positives, while generating Chain-of-Thought style vulnerability reports. Empirical results on real-world datasets show state-of-the-art F1 scores, e.g., $F1_{single}=0.9398$ and $F1_{multi}=0.9330$, with favorable memory and time characteristics. The work offers a scalable, decentralized approach that can aid auditors and practitioners in secure smart-contract deployment.

Abstract

Smart contracts play a significant role in automating blockchain services. Nevertheless, vulnerabilities in smart contracts pose serious threats to blockchain security. Currently, traditional detection methods primarily rely on static analysis and formal verification, which can result in high false-positive rates and poor scalability. Large Language Models (LLMs) have recently made significant progress in smart contract vulnerability detection. However, they still face challenges such as high inference costs and substantial computational overhead. In this paper, we propose ParaVul, a parallel LLM and retrieval-augmented framework to improve the reliability and accuracy of smart contract vulnerability detection. Specifically, we first develop Sparse Low-Rank Adaptation (SLoRA) for LLM fine-tuning. SLoRA introduces sparsification by incorporating a sparse matrix into quantized LoRA-based LLMs, thereby reducing computational overhead and resource requirements while enhancing their ability to understand vulnerability-related issues. We then construct a vulnerability contract dataset and develop a hybrid Retrieval-Augmented Generation (RAG) system that integrates dense retrieval with Best Matching 25 (BM25), assisting in verifying the results generated by the LLM. Furthermore, we propose a meta-learning model to fuse the outputs of the RAG system and the LLM, thereby generating the final detection results. After completing vulnerability detection, we design chain-of-thought prompts to guide LLMs to generate comprehensive vulnerability detection reports. Simulation results demonstrate the superiority of ParaVul, especially in terms of F1 scores, achieving 0.9398 for single-label detection and 0.9330 for multi-label detection.

Figures (5)

• Figure 1: The architecture of ParaVul, which is an intelligent framework for smart contract vulnerability detection that leverages parallel LLMs and RAG. In ParaVul, smart contracts are directly sourced from the blockchain. To ensure detection efficiency, we locally deploy an LLM, the hybrid RAG system, as well as the verification module. Finally, we use API calls to access a cloud-based LLM to generate comprehensive vulnerability detection reports.
• Figure 2: The architecture of SLoRA, which incorporates two adapter modules into each transformer layer: one placed after the projection layer following multi-head attention, and the other after the two fully connected layers. Each adapter consists of a sparse layer and a LoRA layer. The sparse layer masks a subset of nodes, while the LoRA layer applies a low-rank matrix transformation to the data.
• Figure 3: The architecture of the hybrid RAG system, which presents the vulnerability detection process corresponding to the BM25 retrieval strategy and the dense retrieval strategy, respectively.
• Figure 4: Performance evaluation of the verification module in smart contract vulnerability detection. Note that the LLM and RAG detectors take raw smart contracts as input, whereas the verification module operates on the intermediate detection outputs generated by these detectors.
• Figure 5: The generated vulnerability detection report comprises two parts: a detailed analysis of each smart contract vulnerability, describing its location, root cause, associated risks, potential impacts, and recommended mitigation strategies; and a structured summary that consolidates all identified vulnerabilities along with their corresponding solution.