Towards Objective Fine-tuning: How LLMs' Prior Knowledge Causes Potential Poor Calibration?

TL;DR

The paper addresses calibration degradation in fine-tuned LLMs caused by overlaps between pretraining knowledge and downstream data. It introduces CogCalib, a cognition-aware fine-tuning framework that uses an $L_{ m NLL}$-based knowledge-bias detector and adaptive calibration losses (e.g., Label Smoothing, Margin-based Label Smoothing, ECP) with style adaptation to differentiate handling of known versus unknown data. Across 7 tasks and 3 model families, CogCalib consistently improves calibration while preserving performance, achieving an average reduction of $57%$ in $ECE$ on Llama3-8B compared with standard fine-tuning, and it generalizes to out-of-domain tasks. The approach offers practical benefits for domain-specific LLMs in safety-critical human-AI interactions by enhancing objectivity without adding deployment cost, though limitations remain for very large-scale models and calibration strategies may evolve with future calibration terms.

Abstract

Fine-tuned Large Language Models (LLMs) often demonstrate poor calibration, with their confidence scores misaligned with actual performance. While calibration has been extensively studied in models trained from scratch, the impact of LLMs' prior knowledge on calibration during fine-tuning remains understudied. Our research reveals that LLMs' prior knowledge causes potential poor calibration due to the ubiquitous presence of known data in real-world fine-tuning, which appears harmful for calibration. Specifically, data aligned with LLMs' prior knowledge would induce overconfidence, while new knowledge improves calibration. Our findings expose a tension: LLMs' encyclopedic knowledge, while enabling task versatility, undermines calibration through unavoidable knowledge overlaps. To address this, we propose CogCalib, a cognition-aware framework that applies targeted learning strategies according to the model's prior knowledge. Experiments across 7 tasks using 3 LLM families prove that CogCalib significantly improves calibration while maintaining performance, achieving an average 57\% reduction in ECE compared to standard fine-tuning in Llama3-8B. These improvements generalize well to out-of-domain tasks, enhancing the objectivity and reliability of domain-specific LLMs, and making them more trustworthy for critical human-AI interaction applications.

Figures (20)

• Figure 1: LLMs' prior knowledge leads to poor calibration. As LLMs grow stronger, lots of domain-specific fine-tuning data inevitably overlaps with the LLMs' prior knowledge. We reveal that data aligned with the model's prior knowledge (i.e., known data) tend to cause overconfidence, while data exhibiting bias (i.e., unknown data) contribute to better alignment between confidence and accuracy, resulting in more objective predictions.
• Figure 2: Accuracy and ECE of Llama3-8B fine-tuned with different knowledge biases. We fine-tune Llama3-8B using OBQA, with ARC-C and MathQA as OOD tests. The ratio varies from 5:0 to 0:5 (unknown data:known data), with equal dataset sizes. Calibration deteriorates as the knowledge bias lowers, while a higher knowledge bias helps improve calibration.
• Figure 3: (a) Accuracy and confidence of Llama3-8B during fine-tuning on known and unknown data sampled from OBQA. The asymmetric fitting rates between accuracy and confidence in known fine-tuning result in model overconfidence. Conversely, unknown fine-tuning exhibits synchronized fitting of both, minimizing their disparity. (b) Average confidence of correct and incorrect predictions. Unknown fine-tuning yields distinct confidence separation between correct and incorrect samples, facilitating OOD detection.
• Figure 4: Differences in ACC and ECE compared to baseline (delete 0%) under various percentages of known data deletion. The results from in-domain tests indicate that simple bias adjustment fails to achieve consistent calibration improvements across all tasks.
• Figure 5: CogCalib's framework. CogCalib dynamically assesses knowledge bias during training through NLL, employing customized learning strategies with distinct loss functions to enhance calibration. Additionally, CogCalib incorporates a style adaptation process to improve the knowledge bias evaluation performance.