Evaluating and Achieving Controllable Code Completion in Code LLM

TL;DR

This paper introduces $C^3$-Bench, the first instruction-guided benchmark for code completion, designed to evaluate both functional correctness and instruction adherence via ICC and SCC task types across 2,195 Python CCC instances derived from existing datasets. It provides a rigorous benchmark construction pipeline using AST-based middle-code extraction, multiple implementation generations, and automated scoring (Pass@1, IF, ES) with expert validation, revealing gaps between open-source and proprietary models in instruction-following. The authors further present a data-synthesis pipeline that leverages seed demonstrations and automated generation to train $Qwen2.5$-Coder-C$^3$, achieving state-of-the-art results on $C^3$-Bench while preserving performance on conventional benchmarks. Overall, the work highlights substantial gaps in instruction-following for code LLMs, demonstrates the value of automated instruction synthesis for fine-tuning, and provides open resources to spur further research in controllable code generation for open-source models.

Abstract

Code completion has become a central task, gaining significant attention with the rise of large language model (LLM)-based tools in software engineering. Although recent advances have greatly improved LLMs' code completion abilities, evaluation methods have not advanced equally. Most current benchmarks focus solely on functional correctness of code completions based on given context, overlooking models' ability to follow user instructions during completion-a common scenario in LLM-assisted programming. To address this limitation, we present the first instruction-guided code completion benchmark, Controllable Code Completion Benchmark (C3-Bench), comprising 2,195 carefully designed completion tasks. Through comprehensive evaluation of over 40 mainstream LLMs across C3-Bench and conventional benchmarks, we reveal substantial gaps in instruction-following capabilities between open-source and advanced proprietary models during code completion tasks. Moreover, we develop a straightforward data synthesis pipeline that leverages Qwen2.5-Coder to generate high-quality instruction-completion pairs for supervised fine-tuning (SFT). The resulting model, Qwen2.5-Coder-C3, achieves state-of-the-art performance on C3-Bench. Our findings provide valuable insights for enhancing LLMs' code completion and instruction-following capabilities, establishing new directions for future research in code LLMs. To facilitate reproducibility and foster further research in code LLMs, we open-source all code, datasets, and models.

Figures (20)

• Figure 1: Comparison between Controllable Code Completion and traditional code completion tasks. The former extends standard code context by incorporating fine-grained instructions to guide the completion process.
• Figure 2: Overview of the construction pipeline of C$^3$-Bench.
• Figure 3: Training dataset analysis showing (a) sequence length distribution and (b) average component composition (Context Code, Middle Code, and Instruction).
• Figure 4: The ChatML format of C$^3$-bench. < Instruction > indicts the detail control information of completion. < Prefix Code > and < Suffix Code> are the prefix and suffix code.
• Figure 5: Cross-benchmark comparison of model rankings. Base rankings from C$^3$-Bench (leftmost column) are compared with relative ranking changes in existing benchmarks, indicated by color-coded values (green: performance improvement, red: performance degradation, dash: model not evaluated). Spearman correlation coefficients (r) quantify ranking consistency. CCC-32B represents Qwen2.5-Coder-32B-C$^3$.

Theorems & Definitions (2)

• Definition 2.1
• Definition 2.2