Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Case2Code: Scalable Synthetic Data for Code Generation

Yunfan Shao, Linyang Li, Yichuan Ma, Peiji Li, Demin Song, Qinyuan Cheng, Shimin Li, Xiaonan Li, Pengyu Wang, Qipeng Guo, Hang Yan, Xipeng Qiu, Xuanjing Huang, Dahua Lin

TL;DR

Case2Code tackles the challenge of inductive code reasoning by creating a scalable synthetic data pipeline that decouples data quality from a strong teacher model. A writer LLM generates diverse program inputs, a code interpreter executes them to obtain outputs, and the resulting IO triples form Case2Code data that can be used for direct fine-tuning, pre-training, or instruction-tuning. Experiments show zero-shot Case2Code is hard for current LLMs, but training on Case2Code improves both inductive reasoning and general code generation across multiple model sizes and benchmarks like HumanEval and MBPP. The approach demonstrates that large-scale synthetic inductive data can meaningfully boost code reasoning capabilities, offering a cost-effective path when human-labeled data is scarce.

Abstract

Large Language Models (LLMs) have shown outstanding breakthroughs in code generation. Recent work improves code LLMs by training on synthetic data generated by some powerful LLMs, which can be challenging to scale due to the dependence on a teacher model and high generation costs. In this paper, we focus on synthesizing code data at scale and propose a \textbf{Case2Code} task by exploiting the expressiveness and correctness of programs. \textbf{Case2Code} is an inductive inference task that aims to infer underlying code implementations by observing input-output examples or program behaviors, By incorporating LLMs to generate program inputs, and executing the program with these inputs to obtain the program outputs, we can synthesize diverse and high-quality \textbf{Case2Code} data at scale for training and evaluating code LLMs. Experimental results show that case-to-code induction is challenging for current representative LLMs if they are untrained. Models trained with \textbf{Case2Code} improve performance not only on distribution case-to-code induction but also on various coding-generation tasks, demonstrating the great potential of large-scale synthetic data and inductive learning.

Case2Code: Scalable Synthetic Data for Code Generation

TL;DR

Case2Code tackles the challenge of inductive code reasoning by creating a scalable synthetic data pipeline that decouples data quality from a strong teacher model. A writer LLM generates diverse program inputs, a code interpreter executes them to obtain outputs, and the resulting IO triples form Case2Code data that can be used for direct fine-tuning, pre-training, or instruction-tuning. Experiments show zero-shot Case2Code is hard for current LLMs, but training on Case2Code improves both inductive reasoning and general code generation across multiple model sizes and benchmarks like HumanEval and MBPP. The approach demonstrates that large-scale synthetic inductive data can meaningfully boost code reasoning capabilities, offering a cost-effective path when human-labeled data is scarce.

Abstract

Large Language Models (LLMs) have shown outstanding breakthroughs in code generation. Recent work improves code LLMs by training on synthetic data generated by some powerful LLMs, which can be challenging to scale due to the dependence on a teacher model and high generation costs. In this paper, we focus on synthesizing code data at scale and propose a \textbf{Case2Code} task by exploiting the expressiveness and correctness of programs. \textbf{Case2Code} is an inductive inference task that aims to infer underlying code implementations by observing input-output examples or program behaviors, By incorporating LLMs to generate program inputs, and executing the program with these inputs to obtain the program outputs, we can synthesize diverse and high-quality \textbf{Case2Code} data at scale for training and evaluating code LLMs. Experimental results show that case-to-code induction is challenging for current representative LLMs if they are untrained. Models trained with \textbf{Case2Code} improve performance not only on distribution case-to-code induction but also on various coding-generation tasks, demonstrating the great potential of large-scale synthetic data and inductive learning.
Paper Structure (30 sections, 4 figures, 8 tables)

This paper contains 30 sections, 4 figures, 8 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Inductive Inference
  4. Synthetic Data
  5. Method
  6. Problem Formulation
  7. Framework Overview
  8. Collecting Programs
  9. Generating Inputs
  10. Obtain Outputs
  11. Post-processing
  12. Experiment
  13. Experimental Setup
  14. Data Construction
  15. Training Setup
  16. ...and 15 more sections

Figures (4)

  • Figure 1: Examples of existing code generation and case2code tasks. Compared with existing code generation instructions, inductive learning tasks like case2code are rare in the training data, which makes it challenging for LLMs to perform.
  • Figure 2: Our synthetic framework incorporates an LLM and a code interpreter to construct Case2Code training samples at scale automatically.
  • Figure 3: Downstream results when directly fine-tuning InternLM2-7B with different Case2Code prompt templates. Diverse prompts not only help the model to learn Case2Code reasoning but also significantly advance the generalization of the code inductive reasoning.
  • Figure 4: Downstream results when fine-tuning InternLM2-7B with synthetic data using different input example generators. Generator "None" refers to the baseline InternLM2-7B not trained on any Case2Code data. The computational overhead of using LLaMA3-70B is $4.5\times$ that of InternLM2-7B.