Bridging Developer Instructions and Code Completion Through Instruction-Aware Fill-in-the-Middle Paradigm

TL;DR

This work tackles the mismatch between instruction-following and fill-in-the-middle (FIM) code completion by introducing Instruction-aware Fill-in-the-Middle (IFIM). IFIM augments FIM pretraining with an explicit instruction component, learned through a large synthetic dataset where instructions describe the infilled middle, while preserving the original infilling capability. Empirical results on two code LLMs (Deepseek-Coder and Qwen2.5-Coder) show substantial gains in instruction-following (Pass@1 up to ~9 points on HumanEval-infilling and ~2 points on RepoMasterEval) without sacrificing non-instructed completion performance; in fact, IFIM often improves it. The findings establish IFIM as a backward-compatible, practical upgrade for code completion systems, with strong implications for developer productivity and adaptable IDE workflows.

Abstract

Large Language Models (LLMs) have significantly advanced code completion, yet they often fail when the developer's intent is underspecified in the code context. To address this, developers usually add natural language instructions (e.g., comments) into the code context to clarify their intent. However, existing code LLMs applied for code completion systems merely undergo a fill-in-the-middle (FIM) pre-training, which struggles to leverage this information effectively due to the lack of instruction-like training data. Existing instruction-tuning techniques, improving instruction-following in general code generation, paradoxically degrades FIM performance, forcing a trade-off between instruction-following and infilling capabilities. To address this gap, we introduce Instruction-aware Fill-in-the-Middle (IFIM), an instruction-tuning method specifically designed to enhance FIM code completion models. IFIM extends the conventional FIM training objective by incorporating an explicit instruction section into the input, enabling the model to learn from (prefix, instruction, suffix) triplets. This approach allows the model to effectively leverage developer-provided directives while preserving its core completion abilities when no instructions are present. To facilitate this, we constructed a large-scale dataset by using GPT-4o to generate concise, intent-focused instructions for code infilling examples. We evaluated IFIM by applying it to two popular base models, Deepseek-Coder and Qwen2.5-Coder, on the benchmarks derived from HumanEval-infilling and RepoMasterEval. The results demonstrate that IFIM significantly improves instruction-following capabilities, boosting the Pass@1 score from 84.6% to 93.6%. Moreover, this enhancement does not compromise the models' original performance on FIM code completion tasks with no instructions provided.

Figures (4)

• Figure 1: Guiding an LLM for code completion using instructive comments. Left: An ambiguous request leads to an unhelpful completion. Right: A specific comment clarifies the developer's intent, resulting in the desired code.
• Figure 2: The overall framework of IFIM, which consists of synthesizing an instruction dataset and performing an instruction tuning phase.
• Figure 3: A usage example of IFIM-trained code LLMs, where the code context is first propcessed into IFIM format and then infilled by an IFIM-trained code LLM.
• Figure 4: A visual comparison of outputs produced by base and IFIM DeepSeek-Coder given conflicting instructions. The checkmark (✓) indicates that the generated code aligns with the intent of the Conflicting Instruction, while the cross mark (✗) indicates that it does not.