CodeEditorBench: Evaluating Code Editing Capability of Large Language Models

TL;DR

CodeEditorBench introduces a comprehensive, SDLC-aligned benchmark for evaluating LLMs on code editing tasks, spanning debugging, translating, polishing, and requirement switching. It combines a large, diverse dataset with an online judge-based evaluation to provide a rigorous, test-case-driven assessment across 19 models, highlighting performance differences between open- and closed-source LLMs and across problem types. The study reveals that closed-source models generally outperform open-source ones, though results vary by task and prompt, underscoring areas for improvement in code polishing and requirement switching. By releasing prompts and datasets, CodeEditorBench aims to catalyze further research and practical progress in LLM-assisted code editing.

Abstract

Large Language Models (LLMs) for code are rapidly evolving, with code editing emerging as a critical capability. We introduce CodeEditorBench, an evaluation framework designed to rigorously assess the performance of LLMs in code editing tasks, including debugging, translating, polishing, and requirement switching. Unlike existing benchmarks focusing solely on code generation, CodeEditorBench emphasizes real-world scenarios and practical aspects of software development. We curate diverse coding challenges and scenarios from five sources, covering various programming languages, complexity levels, and editing tasks. Evaluation of 19 LLMs reveals that closed-source models (particularly Gemini-Ultra and GPT-4), outperform open-source models in CodeEditorBench, highlighting differences in model performance based on problem types and prompt sensitivities. CodeEditorBench aims to catalyze advancements in LLMs by providing a robust platform for assessing code editing capabilities. We will release all prompts and datasets to enable the community to expand the dataset and benchmark emerging LLMs. By introducing CodeEditorBench, we contribute to the advancement of LLMs in code editing and provide a valuable resource for researchers and practitioners.

Figures (22)

• Figure 1: Overview of CodeEditorBench. CodeEditorBench evaluates programming languages by selecting initial data from five sources and filtering based on code length. It enriches the dataset with Large Language Model-generated test cases, which, along with all code, are verified by an Online Judge System (OJ). The benchmark is developed for four problem types using specific methodologies, described in Section \ref{['method']}. Assessment of 19 LLMs involves crafting prompts for zero-shot, three-shot, and chain of thought settings. Outputs are filtered and integrated with templates for compilation. The OJ's batch judging determines the LLMs' scores, ensuring a rigorous evaluation process.
• Figure 2: Based on Section \ref{['win_rate']}. Left. We propose evaluating LLMs across four scenarios capturing various code editing capabilities, namely code debug, code translate, code polish, and code requirement switch. The figure depicts various model performances across the four scenarios available in CodeEditorBench_Plus in a radial plot – highlighting how relative differences across models change across the scenarios. Right. Performance of open-source and closed-source models on CodeEditorBench_Plus in zero-shot evaluated through win_rate. For a comprehensive explanation of the abbreviation, refer to Section \ref{['subsec:evaluatedmodels']}.
• Figure 3: Data Samples of CodeEditorBench
• Figure 4: Pass rate distribution of models on CodeEditorBench_Plus
• Figure 5: Software Development LifeCycle(Source: https://bigwater.consulting/2019/04/08/software-development-life-cycle-sdlc/)