Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Multicalibration for LLM-based Code Generation

Viola Campos, Robin Kuschnereit, Adrian Ulges

TL;DR

The paper tackles the challenge of calibrating confidence scores for LLM-based code generation by introducing multicalibration that conditions on code-related factors such as complexity, length, and programming language. It evaluates four multicalibration methods across three code benchmarks using open-weight reasoning models, showing substantial improvements in both calibration metrics and practical accuracy over uncalibrated and traditional baseline methods. A key contribution is demonstrating that incorporating structured, task-relevant group information yields significant gains, with language and length emerging as strong signals and all three factors providing complementary benefits. The authors also release CALIBRI, a dataset of code generations with likelihoods and correctness labels to foster further research in code LLM calibration, and outline future directions in broader code-generation tasks and alternative confidence signals.

Abstract

As AI-based code generation becomes widespread, researchers are investigating the calibration of code LLMs - ensuring their confidence scores faithfully represent the true likelihood of code correctness. To do so, we investigate multicalibration, which can capture additional factors about a coding problem, such as complexity, code length, or programming language used. We study four multicalibration approaches on three function synthesis benchmarks, using latest-generation code LLMs (Qwen3 Coder, GPT-OSS, DeepSeek-R1-Distill). Our results demonstrate that multicalibration can yield distinct improvements over both uncalibrated token likelihoods (+1.03 in skill score) and baseline calibrations (+0.37 in skill score). We study the influence of the aforementioned factors in ablations, and make our dataset (consisting of code generations, likelihoods, and correctness labels) available for future research on code LLM calibration.

Multicalibration for LLM-based Code Generation

TL;DR

The paper tackles the challenge of calibrating confidence scores for LLM-based code generation by introducing multicalibration that conditions on code-related factors such as complexity, length, and programming language. It evaluates four multicalibration methods across three code benchmarks using open-weight reasoning models, showing substantial improvements in both calibration metrics and practical accuracy over uncalibrated and traditional baseline methods. A key contribution is demonstrating that incorporating structured, task-relevant group information yields significant gains, with language and length emerging as strong signals and all three factors providing complementary benefits. The authors also release CALIBRI, a dataset of code generations with likelihoods and correctness labels to foster further research in code LLM calibration, and outline future directions in broader code-generation tasks and alternative confidence signals.

Abstract

As AI-based code generation becomes widespread, researchers are investigating the calibration of code LLMs - ensuring their confidence scores faithfully represent the true likelihood of code correctness. To do so, we investigate multicalibration, which can capture additional factors about a coding problem, such as complexity, code length, or programming language used. We study four multicalibration approaches on three function synthesis benchmarks, using latest-generation code LLMs (Qwen3 Coder, GPT-OSS, DeepSeek-R1-Distill). Our results demonstrate that multicalibration can yield distinct improvements over both uncalibrated token likelihoods (+1.03 in skill score) and baseline calibrations (+0.37 in skill score). We study the influence of the aforementioned factors in ablations, and make our dataset (consisting of code generations, likelihoods, and correctness labels) available for future research on code LLM calibration.

Paper Structure

This paper contains 37 sections, 18 equations, 2 figures, 3 tables, 3 algorithms.

Table of Contents

  1. Introduction
  2. Related Work
  3. UE and Calibration for Large Language Models
  4. Post-hoc Calibration
  5. Uncertainty Estimation and Calibration for Code Generation
  6. Background
  7. Calibrated Confidences
  8. Calibration Measures
  9. Brier Score glenn1950verification
  10. Brier Skill Score (BSS) glenn1950verification
  11. Multicalibrated Confidences
  12. Groups
  13. Group-conditional unbiasedness (multiaccuracy)
  14. Group Average Squared Calibration Error (gASCE)
  15. Multicalibration
  16. ...and 22 more sections

Figures (2)

  • Figure 1: Top: Reliability diagrams for Qwen3 Coder on LiveCodeBench. Well calibrated models fit the identity line, while deviations from the diagonal indicate miscalibration. Bin size is encoded by color intensity. Bottom: Mean predicted scores versus accuracies across clustered groups. LINR and IGLB achieve the best calibration, consistent with Table \ref{['tab:results_comparison']}.
  • Figure 2: Ablation experiments evaluating different group combinations on McEval. Results are averaged over the three LLMs.