Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Navigating the Alignment-Calibration Trade-off: A Pareto-Superior Frontier via Model Merging

Tiancheng Hu, Benjamin Minixhofer, Nigel Collier

TL;DR

The paper addresses the alignment tax by showing that post-training calibration degrades alongside performance, leading to overconfident, less reliable outputs. It introduces a simple, post-hoc model merging approach between a pre-trained base and its instruction-tuned version, using a merge coefficient $\lambda \in [0,1]$ to navigate the alignment-calibration frontier. The key finding is the consistent emergence of Pareto-superior interpolations that match or exceed both parents in accuracy while partially restoring calibration, with stronger effects as models scale. This approach offers a practical, computationally cheap path to producing LLMs that are simultaneously more capable and more reliable across model families and merge methods.

Abstract

The "alignment tax" of post-training is typically framed as a drop in task accuracy. We show it also involves a severe loss of calibration, making models overconfident, less reliable, and model outputs less diverse. We show that this trade-off can be navigated effectively via a simple post-hoc intervention: interpolating between a model's weights before and after alignment. Crucially, this is not a strict trade-off. We find that the process consistently reveals Pareto-optimal interpolations - models that improve accuracy beyond both parents while substantially recovering the calibration lost during alignment. Our work demonstrates that simple model merging provides a computationally efficient method for mitigating the full scope of the alignment tax, yielding models that are more capable and more reliable.

Navigating the Alignment-Calibration Trade-off: A Pareto-Superior Frontier via Model Merging

TL;DR

The paper addresses the alignment tax by showing that post-training calibration degrades alongside performance, leading to overconfident, less reliable outputs. It introduces a simple, post-hoc model merging approach between a pre-trained base and its instruction-tuned version, using a merge coefficient to navigate the alignment-calibration frontier. The key finding is the consistent emergence of Pareto-superior interpolations that match or exceed both parents in accuracy while partially restoring calibration, with stronger effects as models scale. This approach offers a practical, computationally cheap path to producing LLMs that are simultaneously more capable and more reliable across model families and merge methods.

Abstract

The "alignment tax" of post-training is typically framed as a drop in task accuracy. We show it also involves a severe loss of calibration, making models overconfident, less reliable, and model outputs less diverse. We show that this trade-off can be navigated effectively via a simple post-hoc intervention: interpolating between a model's weights before and after alignment. Crucially, this is not a strict trade-off. We find that the process consistently reveals Pareto-optimal interpolations - models that improve accuracy beyond both parents while substantially recovering the calibration lost during alignment. Our work demonstrates that simple model merging provides a computationally efficient method for mitigating the full scope of the alignment tax, yielding models that are more capable and more reliable.
Paper Structure (28 sections, 11 figures, 2 tables)

This paper contains 28 sections, 11 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Mitigating Alignment Tax
  4. Model Merging
  5. Experimental Setup
  6. Results and Analysis
  7. The Calibration Cost of Instruction Tuning.
  8. Navigating the Frontier with Model Merging.
  9. General Trend Across Models, Datasets and Merge Methods.
  10. Merging Becomes More Effective and Robust at Scale.
  11. Alignment Induces Calibration Error via Confidence Inflation.
  12. Restoring Calibration via Merging Improves Generative Diversity and Fidelity.
  13. Conclusion
  14. Task Description and Implementation Details
  15. MMLU-Pro
  16. ...and 13 more sections

Figures (11)

  • Figure 1: The alignment-calibration frontier for Gemma-3 models. (Left) IFEval Accuracy vs. MMLU-Pro ECE. (Right) MMLU-Pro Accuracy vs. ECE. Solid lines trace the performance of the PT and IT merges.
  • Figure 2: Performance vs. calibration for the Qwen2.5 model series on (Left) BBH and (Right) MMLU-Pro.
  • Figure 3: Scaling trends for model merging with the Gemma3 family. (a) Peak accuracy gain vs. model size. (b) Normalized accuracy vs. merge coefficient ($\lambda$). (c) Optimal merge coefficient ($\lambda^*$) vs. model size.
  • Figure 4: Mean prediction confidence of PT and IT Gemma3 models of varying sizes on GPQA.
  • Figure 5: The trade-off between generation utility and distinctness on NoveltyBench (curated subset) for Gemma3-12B merges. The path traces merges from the PT to the IT parent.
  • ...and 6 more figures