Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Comments as Natural Logic Pivots: Improve Code Generation via Comment Perspective

Yijie Chen, Yijin Liu, Fandong Meng, Yufeng Chen, Jinan Xu, Jie Zhou

TL;DR

This work argues that code comments serve as natural logic pivots bridging natural language and programming language. It introduces MANGO, which combines a comment-contrastive training objective with a logical comment prompt to improve code generation in small- to mid-sized Code LLMs. Across HumanEval and MBPP benchmarks using backbones like StarCoder and WizardCoder (3B–7B), MANGO yields consistent gains in Pass@k, with notable improvements such as up to 7.52 percentage points on HumanEval Pass@10 and 4.17 on MBPP Pass@10, while the LCP decoding strategy proves more robust than Chain-of-Thought for smaller models. The findings highlight the practicality of leveraging comments to decompose problems and explain code logic, enabling effective code generation without large training costs, and the authors provide code publicly for reproduction.

Abstract

Code generation aims to understand the problem description and generate corresponding code snippets, where existing works generally decompose such complex tasks into intermediate steps by prompting strategies, such as Chain-of-Thought and its variants. While these studies have achieved some success, their effectiveness is highly dependent on the capabilities of advanced Large Language Models (LLMs) such as GPT-4, particularly in terms of API calls, which significantly limits their practical applicability. Consequently, how to enhance the code generation capabilities of small and medium-scale code LLMs without significantly increasing training costs is an appealing challenge. In this paper, we suggest that code comments are the natural logic pivot between natural language and code language and propose using comments to boost the code generation ability of code LLMs. Concretely, we propose MANGO (comMents As Natural loGic pivOts), including a comment contrastive training strategy and a corresponding logical comment decoding strategy. Experiments are performed on HumanEval and MBPP, utilizing StarCoder and WizardCoder as backbone models, and encompassing model parameter sizes between 3B and 7B. The results indicate that MANGO significantly improves the code pass rate based on the strong baselines. Meanwhile, the robustness of the logical comment decoding strategy is notably higher than the Chain-of-thoughts prompting. The code is publicly available at \url{https://github.com/pppa2019/Mango}.

Comments as Natural Logic Pivots: Improve Code Generation via Comment Perspective

TL;DR

This work argues that code comments serve as natural logic pivots bridging natural language and programming language. It introduces MANGO, which combines a comment-contrastive training objective with a logical comment prompt to improve code generation in small- to mid-sized Code LLMs. Across HumanEval and MBPP benchmarks using backbones like StarCoder and WizardCoder (3B–7B), MANGO yields consistent gains in Pass@k, with notable improvements such as up to 7.52 percentage points on HumanEval Pass@10 and 4.17 on MBPP Pass@10, while the LCP decoding strategy proves more robust than Chain-of-Thought for smaller models. The findings highlight the practicality of leveraging comments to decompose problems and explain code logic, enabling effective code generation without large training costs, and the authors provide code publicly for reproduction.

Abstract

Code generation aims to understand the problem description and generate corresponding code snippets, where existing works generally decompose such complex tasks into intermediate steps by prompting strategies, such as Chain-of-Thought and its variants. While these studies have achieved some success, their effectiveness is highly dependent on the capabilities of advanced Large Language Models (LLMs) such as GPT-4, particularly in terms of API calls, which significantly limits their practical applicability. Consequently, how to enhance the code generation capabilities of small and medium-scale code LLMs without significantly increasing training costs is an appealing challenge. In this paper, we suggest that code comments are the natural logic pivot between natural language and code language and propose using comments to boost the code generation ability of code LLMs. Concretely, we propose MANGO (comMents As Natural loGic pivOts), including a comment contrastive training strategy and a corresponding logical comment decoding strategy. Experiments are performed on HumanEval and MBPP, utilizing StarCoder and WizardCoder as backbone models, and encompassing model parameter sizes between 3B and 7B. The results indicate that MANGO significantly improves the code pass rate based on the strong baselines. Meanwhile, the robustness of the logical comment decoding strategy is notably higher than the Chain-of-thoughts prompting. The code is publicly available at \url{https://github.com/pppa2019/Mango}.
Paper Structure (34 sections, 4 equations, 2 figures, 11 tables)

This paper contains 34 sections, 4 equations, 2 figures, 11 tables.

Table of Contents

  1. Introduction
  2. Preliminary Analysis: Token-level Loss of Comment Related Code Styles
  3. MANGO
  4. Backgound: Supervised Fine-tuning on Code Generation Tasks
  5. Comment Contrastive Learning
  6. LCP: Logical Comment Prompt
  7. Experiments and Results
  8. Trainning Settings
  9. SFT (Supervised Fine-Tuning)
  10. CoT (Chain-of-Thought)
  11. LCP (Logical Comment Prompt)
  12. MANGO
  13. Decoding Settings
  14. Evaluation Settings
  15. Evaluate Metrics
  16. ...and 19 more sections

Figures (2)

  • Figure 1: An illustration of the function of code-comment in code LLMs. The comment contributes to breaking down intermediate steps that correspond to the problem description and form an aligned structure with adjacent code lines.
  • Figure 2: The token-level cross-entropy loss of three code styles for a code problem example in the WizardCoder3B model, and the abscissa represents the token index of the tokenized code.