Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Automatic Smart Contract Comment Generation via Large Language Models and In-Context Learning

Junjie Zhao, Xiang Chen, Guang Yang, Yiheng Shen

TL;DR

This paper tackles automatic generation of smart contract comments by introducing SCCLLM, a three-phase framework that harnesses large language models through in-context learning. It pairs a two-stage demonstration retrieval (semantic, then syntactic/lexical) with a carefully constructed prompt to elicit high-quality comments for Solidity code via a ChatGPT API, without model fine-tuning. Empirical results on nearly 30k <method,comment> pairs from 40k+ contracts show that SCCLLM outperforms state-of-the-art baselines in both automatic metrics (BLEU and ROUGE variants) and human judgments, with ablations confirming the value of demonstration selection and the number of demonstrations. The work demonstrates the viability of using LLMs for software documentation tasks in the smart contract domain and points to prompts and demonstrations as critical levers for performance and generalization.

Abstract

The previous smart contract code comment (SCC) generation approaches can be divided into two categories: fine-tuning paradigm-based approaches and information retrieval-based approaches. However, for the fine-tuning paradigm-based approaches, the performance may be limited by the quality of the gathered dataset for the downstream task and they may have knowledge-forgetting issues. While for the information retrieval-based approaches, it is difficult for them to generate high-quality comments if similar code does not exist in the historical repository. Therefore we want to utilize the domain knowledge related to SCC generation in large language models (LLMs) to alleviate the disadvantages of these two types of approaches. In this study, we propose an approach SCCLLM based on LLMs and in-context learning. Specifically, in the demonstration selection phase, SCCLLM retrieves the top-k code snippets from the historical corpus by considering syntax, semantics, and lexical information. In the in-context learning phase, SCCLLM utilizes the retrieved code snippets as demonstrations, which can help to utilize the related knowledge for this task. We select a large corpus from a smart contract community Etherscan.io as our experimental subject. Extensive experimental results show the effectiveness of SCCLLM when compared with baselines in automatic evaluation and human evaluation.

Automatic Smart Contract Comment Generation via Large Language Models and In-Context Learning

TL;DR

This paper tackles automatic generation of smart contract comments by introducing SCCLLM, a three-phase framework that harnesses large language models through in-context learning. It pairs a two-stage demonstration retrieval (semantic, then syntactic/lexical) with a carefully constructed prompt to elicit high-quality comments for Solidity code via a ChatGPT API, without model fine-tuning. Empirical results on nearly 30k <method,comment> pairs from 40k+ contracts show that SCCLLM outperforms state-of-the-art baselines in both automatic metrics (BLEU and ROUGE variants) and human judgments, with ablations confirming the value of demonstration selection and the number of demonstrations. The work demonstrates the viability of using LLMs for software documentation tasks in the smart contract domain and points to prompts and demonstrations as critical levers for performance and generalization.

Abstract

The previous smart contract code comment (SCC) generation approaches can be divided into two categories: fine-tuning paradigm-based approaches and information retrieval-based approaches. However, for the fine-tuning paradigm-based approaches, the performance may be limited by the quality of the gathered dataset for the downstream task and they may have knowledge-forgetting issues. While for the information retrieval-based approaches, it is difficult for them to generate high-quality comments if similar code does not exist in the historical repository. Therefore we want to utilize the domain knowledge related to SCC generation in large language models (LLMs) to alleviate the disadvantages of these two types of approaches. In this study, we propose an approach SCCLLM based on LLMs and in-context learning. Specifically, in the demonstration selection phase, SCCLLM retrieves the top-k code snippets from the historical corpus by considering syntax, semantics, and lexical information. In the in-context learning phase, SCCLLM utilizes the retrieved code snippets as demonstrations, which can help to utilize the related knowledge for this task. We select a large corpus from a smart contract community Etherscan.io as our experimental subject. Extensive experimental results show the effectiveness of SCCLLM when compared with baselines in automatic evaluation and human evaluation.
Paper Structure (26 sections, 5 equations, 9 figures, 5 tables)

This paper contains 26 sections, 5 equations, 9 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Our Proposed Approach
  3. Demonstration Selection Phase
  4. Semantic-based Retrieval Part
  5. Syntax and Lexical-based Retrieval Part
  6. In-Context Learning Phase
  7. LLMs Inference Phase
  8. Experimental Setup
  9. Research Questions
  10. Experimental Subject
  11. Performance Measures
  12. Baselines
  13. Implementation Details
  14. Running Platform
  15. Result Analysis
  16. ...and 11 more sections

Figures (9)

  • Figure 1: Framework of our proposed approach SCCLLM
  • Figure 2: The prompt template used by our proposed approach SCCLLM
  • Figure 3: Examples of two problems in the dataset shared by Yang et al. yang2021multi.
  • Figure 4: Comments generated by SCCLLM and baselines for two cases
  • Figure 5: A questionnaire used in our human study
  • ...and 4 more figures