Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Increasing LLM Coding Capabilities through Diverse Synthetic Coding Tasks

Amal Abed, Ivan Lukic, Jörg K. H. Franke, Frank Hutter

TL;DR

The paper addresses the shortage of diverse, reasoning-rich data for coding with LLMs by proposing a scalable pipeline that generates nearly 800k instruction–reasoning–code–test quadruplets. It combines curated seeds, web-mined content, reasoning-guided expansion, and execution-based validation, enhanced by a genetic-instruct mechanism to broaden task coverage. Fine-tuning a 2.7B Phi-2 model on this data yields consistent improvements on HumanEval and MBPP, and gains transfer across architectures while achieving efficiency advantages over larger models. The results underscore that dataset design emphasizing reasoning and diversity can rival scaling up model size and open pathways to more accessible, capable code-generation systems. The authors also discuss limitations (Python-only) and future directions toward multilingual programming and pretraining-scale deployment.

Abstract

Large language models (LLMs) have shown impressive promise in code generation, yet their progress remains limited by the shortage of large-scale datasets that are both diverse and well-aligned with human reasoning. Most existing resources pair problems with solutions, but omit the intermediate thought process that guides coding. To close this gap, we present a scalable synthetic data generation pipeline that produces nearly 800k instruction-reasoning-code-test quadruplets. Each sample combines a task, a step-by-step reasoning trace, a working solution, and executable tests, enabling models to learn not just the what but also the how of problem solving. Our pipeline combines four key components: curated contest problems, web-mined content filtered by relevance classifiers, data expansion guided by reasoning patterns, and multi-stage execution-based validation. A genetic mutation algorithm further increases task diversity while maintaining consistency between reasoning traces and code implementations. Our key finding is that fine-tuning LLMs on this dataset yields consistent improvements on coding benchmarks. Beyond raw accuracy, reasoning-aware data can substitute for model scaling, generalize across architectures, and outperform leading open-source alternatives under identical sample budgets. Our work establishes reasoning-centered synthetic data generation as an efficient approach for advancing coding capabilities in LLMs. We publish our dataset and generation pipeline to facilitate further research.

Increasing LLM Coding Capabilities through Diverse Synthetic Coding Tasks

TL;DR

The paper addresses the shortage of diverse, reasoning-rich data for coding with LLMs by proposing a scalable pipeline that generates nearly 800k instruction–reasoning–code–test quadruplets. It combines curated seeds, web-mined content, reasoning-guided expansion, and execution-based validation, enhanced by a genetic-instruct mechanism to broaden task coverage. Fine-tuning a 2.7B Phi-2 model on this data yields consistent improvements on HumanEval and MBPP, and gains transfer across architectures while achieving efficiency advantages over larger models. The results underscore that dataset design emphasizing reasoning and diversity can rival scaling up model size and open pathways to more accessible, capable code-generation systems. The authors also discuss limitations (Python-only) and future directions toward multilingual programming and pretraining-scale deployment.

Abstract

Large language models (LLMs) have shown impressive promise in code generation, yet their progress remains limited by the shortage of large-scale datasets that are both diverse and well-aligned with human reasoning. Most existing resources pair problems with solutions, but omit the intermediate thought process that guides coding. To close this gap, we present a scalable synthetic data generation pipeline that produces nearly 800k instruction-reasoning-code-test quadruplets. Each sample combines a task, a step-by-step reasoning trace, a working solution, and executable tests, enabling models to learn not just the what but also the how of problem solving. Our pipeline combines four key components: curated contest problems, web-mined content filtered by relevance classifiers, data expansion guided by reasoning patterns, and multi-stage execution-based validation. A genetic mutation algorithm further increases task diversity while maintaining consistency between reasoning traces and code implementations. Our key finding is that fine-tuning LLMs on this dataset yields consistent improvements on coding benchmarks. Beyond raw accuracy, reasoning-aware data can substitute for model scaling, generalize across architectures, and outperform leading open-source alternatives under identical sample budgets. Our work establishes reasoning-centered synthetic data generation as an efficient approach for advancing coding capabilities in LLMs. We publish our dataset and generation pipeline to facilitate further research.
Paper Structure (19 sections, 4 figures, 3 tables)

This paper contains 19 sections, 4 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Methodology
  4. Dataset Curation and Expansion
  5. Structuring into Instruction–Reasoning–Solution–Test Quadruplets
  6. Execution-Based Validation and Refinement
  7. Evolutionary Expansion with Genetic-Instruct
  8. Deduplication, Diversity Preservation, and Decontamination
  9. Fine-Tuning Setup
  10. Evaluation
  11. Results
  12. HumanEval and MBPP Benchmarks
  13. Efficiency Gains
  14. Experiments
  15. Cross-Model Generalization
  16. ...and 4 more sections

Figures (4)

  • Figure 1: Process Flow for Genetic Instruct
  • Figure 2: Pass rates on HumanEval and MBPP benchmarks for selected models.
  • Figure 3: Pass@1 scores on HumanEval and MBPP for homogeneous vs. diverse subsets. Diversity yields stronger reasoning alignment.
  • Figure 4: Comparison of pass@1 scores on HumanEval and MBPP for models fine-tuned on 5k samples from our dataset versus EpiCoder-func-380k and Self-OSS-Instruct-SC2-Exec-Filter-50k.