Beyond Code Similarity: Benchmarking the Plausibility, Efficiency, and Complexity of LLM-Generated Smart Contracts

TL;DR

The paper investigates how well current LLMs can generate Solidity Smart Contracts by evaluating semantic similarity, structural fidelity, functional plausibility, gas efficiency, and maintainability. It contrasts zero-shot generation with retrieval-augmented generation (RAG) across four models on 500 real-world function examples, using automated testing and Solidity-specific complexity probes. Key findings show high semantic and structural alignment yet limited functional plausibility, with RAG substantially boosting correctness and efficiency in several models, notably DeepSeek-Coder-v2, while revealing trade-offs in complexity. The work highlights the potential of open, RAG-enabled pipelines for production-adjacent code generation but emphasizes the need for rigorous expert validation to ensure secure and correct contracts.

Abstract

Smart Contracts are critical components of blockchain ecosystems, with Solidity as the dominant programming language. While LLMs excel at general-purpose code generation, the unique constraints of Smart Contracts, such as gas consumption, security, and determinism, raise open questions about the reliability of LLM-generated Solidity code. Existing studies lack a comprehensive evaluation of these critical functional and non-functional properties. We benchmark four state-of-the-art models under zero-shot and retrieval-augmented generation settings across 500 real-world functions. Our multi-faceted assessment employs code similarity metrics, semantic embeddings, automated test execution, gas profiling, and cognitive and cyclomatic complexity analysis. Results show that while LLMs produce code with high semantic similarity to real contracts, their functional correctness is low: only 20% to 26% of zero-shot generations behave identically to ground-truth implementations under testing. The generated code is consistently simpler, with significantly lower complexity and gas consumption, often due to omitted validation logic. Retrieval-Augmented Generation markedly improves performance, boosting functional correctness by up to 45% and yielding more concise and efficient code. Our findings reveal a significant gap between semantic similarity and functional plausibility in LLM-generated Smart Contracts. We conclude that while RAG is a powerful enhancer, achieving robust, production-ready code generation remains a substantial challenge, necessitating careful expert validation.

Figures (9)

• Figure 1: Example of a function whose syntax is disrupted in the original dataset.
• Figure 2: Prompt structure used for baseline code generation.
• Figure 3: Prompt structure used for generation with RAG.
• Figure 4: Prompt used for constructor injection.
• Figure 5: Dependency declaration example.