Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

From Guessing to Placeholding: A Cost-Theoretic Framework for Uncertainty-Aware Code Completion

Liang Zhu, Haolin Chen, Lidong Zhao, Xian Wu

Abstract

While Large Language Models (LLMs) have demonstrated exceptional proficiency in code completion, they typically adhere to a Hard Completion (HC) paradigm, compelling the generation of fully concrete code even amidst insufficient context. Our analysis of 3 million real-world interactions exposes the limitations of this strategy: 61% of the generated suggestions were either edited after acceptance or rejected despite exhibiting over 80% similarity to the user's subsequent code, suggesting that models frequently make erroneous predictions at specific token positions. Motivated by this observation, we propose Adaptive Placeholder Completion (APC), a collaborative framework that extends HC by strategically outputting explicit placeholders at high-entropy positions, allowing users to fill directly via IDE navigation. Theoretically, we formulate code completion as a cost-minimization problem under uncertainty. Premised on the observation that filling placeholders incurs lower cost than correcting errors, we prove the existence of a critical entropy threshold above which APC achieves strictly lower expected cost than HC. We instantiate this framework by constructing training data from filtered real-world edit logs and design a cost-based reward function for reinforcement learning. Extensive evaluations across 1.5B--14B parameter models demonstrate that APC reduces expected editing costs from 19% to 50% while preserving standard HC performance. Our work provides both a theoretical foundation and a practical training framework for uncertainty-aware code completion, demonstrating that adaptive abstention can be learned end-to-end without sacrificing conventional completion quality.

From Guessing to Placeholding: A Cost-Theoretic Framework for Uncertainty-Aware Code Completion

Abstract

While Large Language Models (LLMs) have demonstrated exceptional proficiency in code completion, they typically adhere to a Hard Completion (HC) paradigm, compelling the generation of fully concrete code even amidst insufficient context. Our analysis of 3 million real-world interactions exposes the limitations of this strategy: 61% of the generated suggestions were either edited after acceptance or rejected despite exhibiting over 80% similarity to the user's subsequent code, suggesting that models frequently make erroneous predictions at specific token positions. Motivated by this observation, we propose Adaptive Placeholder Completion (APC), a collaborative framework that extends HC by strategically outputting explicit placeholders at high-entropy positions, allowing users to fill directly via IDE navigation. Theoretically, we formulate code completion as a cost-minimization problem under uncertainty. Premised on the observation that filling placeholders incurs lower cost than correcting errors, we prove the existence of a critical entropy threshold above which APC achieves strictly lower expected cost than HC. We instantiate this framework by constructing training data from filtered real-world edit logs and design a cost-based reward function for reinforcement learning. Extensive evaluations across 1.5B--14B parameter models demonstrate that APC reduces expected editing costs from 19% to 50% while preserving standard HC performance. Our work provides both a theoretical foundation and a practical training framework for uncertainty-aware code completion, demonstrating that adaptive abstention can be learned end-to-end without sacrificing conventional completion quality.

Paper Structure

This paper contains 41 sections, 2 theorems, 40 equations, 2 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Code Completion Paradigms
  4. Uncertainty Estimation and Selective Generation
  5. Reinforcement Learning for LLMs
  6. Theoretical Framework
  7. Problem Formulation
  8. Expected Cost Analysis
  9. Optimality Condition for APC
  10. From Theory to Practice
  11. Experiments
  12. Experimental Setup
  13. Datasets
  14. Training
  15. Evaluation
  16. ...and 26 more sections

Key Result

Theorem 3.3

Given Assumption ass:cost_inequality, there exists a unique critical entropy threshold: such that for any token position $i$ in the completion sequence where the conditional entropy $H(y_i^* \mid c, y_{<i}^*)$ exceeds $H^*$, the Adaptive Placeholder Completion strategy achieves strictly lower expected cost than Hard Completion: $\blacktriangleleft$$\blacktriangleleft$

Figures (2)

  • Figure 1: Fixed-threshold post-processing on base Qwen2.5-Coder-Instruct models. (a) Entropy threshold sweeps and (b) confidence threshold sweeps both show that Cost converges to the base model level (dashed line) regardless of threshold values, demonstrating that explicit thresholds cannot effectively reduce cost without implicit learning.
  • Figure 2: Cumulative average entropy for PC regions vs. HC regions. PC regions show substantially higher entropy accumulation, confirming that human-annotated placeholders align with high-uncertainty positions.

Theorems & Definitions (3)

  • Remark 3.2
  • Theorem 3.3
  • Corollary 3.4