Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Generating High-Quality Datasets for Code Editing via Open-Source Language Models

Zekai Zhang, Mingwei Liu, Zhenxi Chen, Linxi Liang, Yuxuan Chen, Guangsheng Ou, Yanlin Wang, Dan Li, Xin Peng, Zibin Zheng

TL;DR

OpenCodeEdit presents an open-source pipeline to synthesize high-quality code-editing data using two complementary open LLMs, generating both lazy and descriptive instructions and a diverse pre-edit/post-edit triplet format. The resulting OCEDataFT dataset (20k samples) enables efficient instruction-tuning, yielding substantial pass@1 improvements across multiple base models and narrowing the gap to GPT-4 without relying on proprietary data. Core innovations include a dual-LLM synthesis loop, a two-stage DT-Filtering framework (diff and topic-based filtering), and multi-LLM data integration that enhances task and linguistic diversity. The work demonstrates the value of task-aligned synthetic data for code editing, provides a reproducible pipeline and artifacts, and offers practical guidance for enterprise and research settings aiming to improve code-editing capabilities with open resources.

Abstract

Code editing plays a vital role in software engineering, requiring developers to adjust existing code according to natural language instructions while keeping functionality intact and avoiding unnecessary modifications. However, commit-based datasets commonly used for this task are often noisy, lack diversity, and fail to reflect the style of real-world edit instructions. To address this, we introduce OpenCodeEdit, an open-source pipeline that leverages multiple LLMs to synthesize realistic code-edit triplets. The pipeline produces both concise "lazy" instructions and more detailed "descriptive" ones, and applies filtering based on diffs and topics to guarantee data quality and variety. Using this process, we construct OCEDataFT, a curated dataset of 20K samples. Fine-tuning three advanced base models on OCEDataFT leads to significant performance boosts on the CanItEdit benchmark, with relative pass@1 improvements ranging from 4.50% to 20.79%. Notably, the resulting models achieve performance close to closed-source systems, narrowing the gap to GPT-4 to just 3.54%, without relying on proprietary resources or manual annotation.

Generating High-Quality Datasets for Code Editing via Open-Source Language Models

TL;DR

OpenCodeEdit presents an open-source pipeline to synthesize high-quality code-editing data using two complementary open LLMs, generating both lazy and descriptive instructions and a diverse pre-edit/post-edit triplet format. The resulting OCEDataFT dataset (20k samples) enables efficient instruction-tuning, yielding substantial pass@1 improvements across multiple base models and narrowing the gap to GPT-4 without relying on proprietary data. Core innovations include a dual-LLM synthesis loop, a two-stage DT-Filtering framework (diff and topic-based filtering), and multi-LLM data integration that enhances task and linguistic diversity. The work demonstrates the value of task-aligned synthetic data for code editing, provides a reproducible pipeline and artifacts, and offers practical guidance for enterprise and research settings aiming to improve code-editing capabilities with open resources.

Abstract

Code editing plays a vital role in software engineering, requiring developers to adjust existing code according to natural language instructions while keeping functionality intact and avoiding unnecessary modifications. However, commit-based datasets commonly used for this task are often noisy, lack diversity, and fail to reflect the style of real-world edit instructions. To address this, we introduce OpenCodeEdit, an open-source pipeline that leverages multiple LLMs to synthesize realistic code-edit triplets. The pipeline produces both concise "lazy" instructions and more detailed "descriptive" ones, and applies filtering based on diffs and topics to guarantee data quality and variety. Using this process, we construct OCEDataFT, a curated dataset of 20K samples. Fine-tuning three advanced base models on OCEDataFT leads to significant performance boosts on the CanItEdit benchmark, with relative pass@1 improvements ranging from 4.50% to 20.79%. Notably, the resulting models achieve performance close to closed-source systems, narrowing the gap to GPT-4 to just 3.54%, without relying on proprietary resources or manual annotation.

Paper Structure

This paper contains 39 sections, 1 equation, 8 figures, 7 tables.

Table of Contents

  1. Introduction
  2. Methodology
  3. Code Editing Dataset Format
  4. Seed Code Snippet Extraction
  5. Pre-edit Code and Edit Instruction Generation
  6. Post-edit Code Generation
  7. Data Filtering
  8. Diff Filtering
  9. Topic Filtering
  10. Summary of DT-Filtering
  11. Implementation Details
  12. Seed Code Snippet Extraction
  13. Data Synthesis
  14. Data Filtering
  15. Model Training
  16. ...and 24 more sections

Figures (8)

  • Figure 1: Overview of OpenCodeEdit.
  • Figure 2: An example of the pre-edit code, post-edit code, and the edit instructions.
  • Figure 3: An example of the two-round dialogue.
  • Figure 4: Distribution of modified lines, hunks, and topics in the combined dataset from Qwen3 and DeepSeek.
  • Figure 5: The input-output format of model training and evaluation.
  • ...and 3 more figures