Success is in the Details: Evaluate and Enhance Details Sensitivity of Code LLMs through Counterfactuals

TL;DR

The paper identifies a critical blind spot in Code LLM evaluation: sensitivity to subtle changes in problem details. It introduces the CTF-Code benchmark to systematically probe this sensitivity via counterfactual problem perturbations, and the CTF-Instruct framework to generate sensitivity-focused data with a three-dimensional completion (difficulty, diversity, sensitivity). Empirical results show that many state-of-the-art Code LLMs exhibit significant drops on CTF-Code, while finetuning with CTF-Instruct data yields meaningful improvements across CTF-Code and broader benchmarks, validating sensitivity optimization as a practical path to more robust code understanding. Overall, the work expands the evaluation paradigm for Code LLMs and provides scalable methods to enhance their sensitivity, with implications for reliability and cross-domain generalization in code tasks.

Abstract

Code Sensitivity refers to the ability of Code LLMs to recognize and respond to details changes in problem descriptions. While current code benchmarks and instruction data focus on difficulty and diversity, sensitivity is overlooked. We first introduce the CTF-Code benchmark, constructed using counterfactual perturbations, minimizing input changes while maximizing output changes. The evaluation shows that many LLMs have a more than 10\% performance drop compared to the original problems. To fully utilize sensitivity, CTF-Instruct, an incremental instruction fine-tuning framework, extends on existing data and uses a selection mechanism to meet the three dimensions of difficulty, diversity, and sensitivity. Experiments show that LLMs fine-tuned with CTF-Instruct data achieve over a 2\% improvement on CTF-Code, and more than a 10\% performance boost on LiveCodeBench, validating the feasibility of enhancing LLMs' sensitivity to improve performance.

Figures (16)

• Figure 1: While diversity and difficulty have been explored, sensitivity to problem details remains underexplored. In the original problem, the smallest number should be increased, whereas in the counterfactual version, no matter which number is modified, the result remains the same—double the cumulative sum.
• Figure 2: The pipeline of CTF-Code benchmark construction. First, original problems are sent to LLMs to sample semantic permutations on the problem description. Algorithmic experts will carefully check the CTF problems and decide to drop or fix them to generate CTF solutions. After selecting the most suitable CTF problem, its testcases are constructed by executing its solution on the inputs from the original testcases.
• Figure 3: The evaluation results of Code LLMs on CTF-Code.
• Figure 4: The change in model performance as sensitivity data joined into Evol-Instruct.
• Figure 5: The performance change brought by the selection strategy on Evol-Instruct and Oss-Instruct. The darker shade in Humaneval represents Humaneval+, while in LCB, the darker shade represents LCB-All, and the lighter shade represents LCB-Easy.