InstructCoder: Instruction Tuning Large Language Models for Code Editing

TL;DR

This work addresses the challenge of automatic code editing by introducing InstructCoder, a large, instruction-tuning dataset crafted to train LLMs to perform diverse code edits under natural-language instructions. It pairs seed GitHub commit data with machine-generated instructions and scenario-conditioned generation, processed through Rouge-L and MinHash-LSH to yield over 114k high-quality tasks. A new execution-based benchmark, EditEval, rigorously evaluates code-editing ability across diverse tasks, revealing that open-source models can reach or approach the performance of proprietary systems when fine-tuned with InstructCoder. The results highlight the importance of base-model pretraining and data scale, and the work contributes practical resources and a scalable pathway toward improved AI-assisted code editing for developers.

Abstract

Code editing encompasses a variety of pragmatic tasks that developers deal with daily. Despite its relevance and practical usefulness, automatic code editing remains an underexplored area in the evolution of deep learning models, partly due to data scarcity. In this work, we explore the use of Large Language Models (LLMs) to edit code based on user instructions. Evaluated on a novel human-written execution-based benchmark dubbed EditEval, we found current models often struggle to fulfill the instructions. In light of this, we contribute InstructCoder, the first instruction-tuning dataset designed to adapt LLMs for general-purpose code editing, containing high-diversity code-editing tasks such as comment insertion, code optimization, and code refactoring. It consists of over 114,000 instruction-input-output triplets and covers multiple distinct code editing scenarios. The collection process starts with filtered commit data sourced from GitHub Python repositories as seeds. Subsequently, the dataset is systematically expanded through an iterative process, where both seed and generated tasks are used to prompt ChatGPT for more data. Our findings reveal that open-source LLMs fine-tuned on InstructCoder can significantly enhance the accuracy of code edits, exhibiting superior code-editing performance matching advanced proprietary LLMs. The datasets and the source code are publicly available at https://github.com/qishenghu/CodeInstruct.

Figures (14)

• Figure 1: Data collection pipeline of InstructCoder (left) and a qualitative example from the dataset (right, best viewed with zoom). Initial seed tasks are selected from GitHub commits, and inspire ChatGPT to generate new instructions. Plausible scenarios where the filtered instructions may be used are then generated. Finally, corresponding code input and output are obtained conditioned on both the instruction and scenario. High-quality samples are manually selected and recurrently added to the task pool for further generation.
• Figure 2: Distribution of code edit intent categories.
• Figure 3: Visualizations of InstructCoder data. Best viewed in zoom.
• Figure 4: Token length distribution of InstructCoder
• Figure 5: Data scaling performance of InstructCoder on LLaMA evaluated on EditEval, using 1%, 10% and 100% training data.