Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Semi-Instruct: Bridging Natural-Instruct and Self-Instruct for Code Large Language Models

Xianzhen Luo, Qingfu Zhu, Zhiming Zhang, Xu Wang, Qing Yang, Dongliang Xu, Wanxiang Che

TL;DR

Semi-Instruct addresses the limitations of two dominant Code LLM tuning paradigms, NI and SI, by generating diverse yet proper instruction-code pairs from NI while validating correctness through test-case construction that executes original code. The method comprises generation, validation, and ranking steps, enabling a curriculum-like training signal based on the number of test cases passed. Empirical results on HumanEval show that SemI outperforms both NI and SI, and that combining SemI with SI or NI yields further gains with data scale, indicating robust improvements in program synthesis. Overall, Semi-Instruct provides a scalable, data-efficient approach to instruction tuning for code LLMs with practical impact on code generation quality and reliability.

Abstract

Instruction tuning plays a pivotal role in Code Large Language Models (Code LLMs) for the task of program synthesis. Presently, two dominant paradigms for collecting tuning data are natural-instruct (human-written) and self-instruct (automatically generated). Natural-instruct includes diverse and correct codes but lacks instruction-code pairs, and exists improper code formats like nested single-line codes. In contrast, self-instruct automatically generates proper paired data. However, it suffers from low diversity due to generating duplicates and cannot ensure the correctness of codes. To bridge the both paradigms, we propose \textbf{Semi-Instruct}. It first converts diverse but improper codes from natural-instruct into proper instruction-code pairs through a method similar to self-instruct. To verify the correctness of generated codes, we design a novel way to construct test cases by generating cases' inputs and executing correct codes from natural-instruct to get outputs. Finally, diverse and correct instruction-code pairs are retained for instruction tuning. Experiments show that semi-instruct is significantly better than natural-instruct and self-instruct. Furthermore, the performance steadily improves as data scale increases.

Semi-Instruct: Bridging Natural-Instruct and Self-Instruct for Code Large Language Models

TL;DR

Semi-Instruct addresses the limitations of two dominant Code LLM tuning paradigms, NI and SI, by generating diverse yet proper instruction-code pairs from NI while validating correctness through test-case construction that executes original code. The method comprises generation, validation, and ranking steps, enabling a curriculum-like training signal based on the number of test cases passed. Empirical results on HumanEval show that SemI outperforms both NI and SI, and that combining SemI with SI or NI yields further gains with data scale, indicating robust improvements in program synthesis. Overall, Semi-Instruct provides a scalable, data-efficient approach to instruction tuning for code LLMs with practical impact on code generation quality and reliability.

Abstract

Instruction tuning plays a pivotal role in Code Large Language Models (Code LLMs) for the task of program synthesis. Presently, two dominant paradigms for collecting tuning data are natural-instruct (human-written) and self-instruct (automatically generated). Natural-instruct includes diverse and correct codes but lacks instruction-code pairs, and exists improper code formats like nested single-line codes. In contrast, self-instruct automatically generates proper paired data. However, it suffers from low diversity due to generating duplicates and cannot ensure the correctness of codes. To bridge the both paradigms, we propose \textbf{Semi-Instruct}. It first converts diverse but improper codes from natural-instruct into proper instruction-code pairs through a method similar to self-instruct. To verify the correctness of generated codes, we design a novel way to construct test cases by generating cases' inputs and executing correct codes from natural-instruct to get outputs. Finally, diverse and correct instruction-code pairs are retained for instruction tuning. Experiments show that semi-instruct is significantly better than natural-instruct and self-instruct. Furthermore, the performance steadily improves as data scale increases.
Paper Structure (33 sections, 4 figures, 2 tables)

This paper contains 33 sections, 4 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Code LLM
  4. Code Instruction Tuning
  5. Test Cases
  6. Methodology
  7. Generation
  8. Validation
  9. Ranking
  10. Experiments
  11. Dataset Construction
  12. Natural-Instruct Dataset
  13. Self-Instruct Dataset
  14. Semi-Instruct Dataset
  15. Data Selection & Order
  16. ...and 18 more sections

Figures (4)

  • Figure 1: The advantages and disadvantages of natural-instruct and self-instruct. Their colors correspond to the data samples.
  • Figure 2: Pipeline of Semi-Instruct. It has three main components. (1) Generation: Given the original codes, generate instructions, a fixed number of test cases' inputs based on the instructions, and refined codes; (2) Validation: Run the original codes on the inputs, obtain complete test cases through extracting outputs from those successful executions, and subsequently retain refined codes that pass all test cases; (3) Ranking: Since the more difficult the instructions are, the less test cases are constructed, sort the data in reverse order according to the number of test cases.
  • Figure 3: p@1 results on the HumanEval dataset. Note that unlike self-instruct and semi-instruct, the total amount of data for semi-instruct is only 40k.
  • Figure 4: Code generated for the same problem after training the model using the natural-instruct dataset and the semi-instruct dataset, respectively. The former is wrong, and the latter is correct.