Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Improving Natural Language Capability of Code Large Language Model

Wei Li, Daoguang Zan, Bei Guan, Ailun Yu, Xiaolin Chen, Yongji Wang

TL;DR

This work addresses the gap in natural language understanding within code large language models by introducing a two-module framework, AttentionExtractor and AttentionCoder, that extracts key NL phrases from task descriptions and uses them to guide code generation. The approach is formalized as $\mathcal{A}=\mathcal{M}_E(\mathbf{x})$ and $\mathbf{y}=\mathcal{M}_C(\mathbf{x}, \mathcal{A})$, and is evaluated on the multilingual MultiNL-H benchmark derived from HumanEval across English, Chinese, French, Spanish, and Japanese. Results show consistent gains over baselines across models, with notable improvements such as $>10\%$ absolute in Chinese GPT-3.5-turbo on $pass@1$, and human-in-the-loop enhancements further boosting performance. The framework also demonstrates transferability to code generation in multiple programming languages, code translation, and mathematical reasoning, indicating practical benefits for improving NL-driven code tasks while aligning with public data and reproducibility goals.

Abstract

Code large language models (Code LLMs) have demonstrated remarkable performance in code generation. Nonetheless, most existing works focus on boosting code LLMs from the perspective of programming capabilities, while their natural language capabilities receive less attention. To fill this gap, we thus propose a novel framework, comprising two modules: AttentionExtractor, which is responsible for extracting key phrases from the user's natural language requirements, and AttentionCoder, which leverages these extracted phrases to generate target code to solve the requirement. This framework pioneers an innovative idea by seamlessly integrating code LLMs with traditional natural language processing tools. To validate the effectiveness of the framework, we craft a new code generation benchmark, called MultiNL-H, covering five natural languages. Extensive experimental results demonstrate the effectiveness of our proposed framework.

Improving Natural Language Capability of Code Large Language Model

TL;DR

This work addresses the gap in natural language understanding within code large language models by introducing a two-module framework, AttentionExtractor and AttentionCoder, that extracts key NL phrases from task descriptions and uses them to guide code generation. The approach is formalized as and , and is evaluated on the multilingual MultiNL-H benchmark derived from HumanEval across English, Chinese, French, Spanish, and Japanese. Results show consistent gains over baselines across models, with notable improvements such as absolute in Chinese GPT-3.5-turbo on , and human-in-the-loop enhancements further boosting performance. The framework also demonstrates transferability to code generation in multiple programming languages, code translation, and mathematical reasoning, indicating practical benefits for improving NL-driven code tasks while aligning with public data and reproducibility goals.

Abstract

Code large language models (Code LLMs) have demonstrated remarkable performance in code generation. Nonetheless, most existing works focus on boosting code LLMs from the perspective of programming capabilities, while their natural language capabilities receive less attention. To fill this gap, we thus propose a novel framework, comprising two modules: AttentionExtractor, which is responsible for extracting key phrases from the user's natural language requirements, and AttentionCoder, which leverages these extracted phrases to generate target code to solve the requirement. This framework pioneers an innovative idea by seamlessly integrating code LLMs with traditional natural language processing tools. To validate the effectiveness of the framework, we craft a new code generation benchmark, called MultiNL-H, covering five natural languages. Extensive experimental results demonstrate the effectiveness of our proposed framework.
Paper Structure (30 sections, 12 figures, 3 tables)

This paper contains 30 sections, 12 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Framework
  3. Methodology
  4. AttentionExtractor
  5. AttentionCoder
  6. Benchmark Construction
  7. Experiments
  8. Experimental Setup
  9. Benchmark
  10. Baselines
  11. Evaluation Metrics
  12. Implementation Details
  13. Effectiveness Validation
  14. Ablation Study
  15. Attention Granularity Levels
  16. ...and 15 more sections

Figures (12)

  • Figure 1: Pass$@1$ of OpenAI's GPT-3.5-turbo evaluated on HumanEval across multiple natural languages.
  • Figure 2: The overview of our proposed framework.
  • Figure 3: The implementation details of our framework: AttentionExtractor and AttentionCoder.
  • Figure 4: The result of influence factors on GPT-3.5: Attention Granularity Levels, Ranking Algorithms, Attention Count; "Top-x" in (c) means providing LLMs with x key phrases extracted by AttentionExtractor, "Top-0" donates baseline (No Attention), "All" means providing LLMs all extracted Attention.
  • Figure 5: The results of model size: ordinate value is the average of five natural language results on Pass$@1$.
  • ...and 7 more figures