Annotation-Efficient Universal Honesty Alignment

TL;DR

This work tackles the challenge of universal honesty alignment in large language models by proposing EliCal, a two-stage, annotation-efficient framework that first elicits internal confidence through self-consistency (Stage 1) and then calibrates this confidence with a small set of correctness annotations (Stage 2). It freezes the backbone and uses LoRA adapters plus a final linear head to predict confidence, enabling pre-generation honesty estimates and reducing the need for costly labeling. To support large-scale study, the authors introduce HonestyBench, a ten-dataset benchmark with over 560k training examples and 71k evaluation instances, annotated for correctness and semantic consistency. Empirical results show EliCal achieves about 98% of the upper-bound performance set by full supervision and outperforms calibration-only baselines on unseen tasks like MMLU, demonstrating strong generalization and practical potential for scalable honesty alignment in real-world AI systems.

Abstract

Honesty alignment-the ability of large language models (LLMs) to recognize their knowledge boundaries and express calibrated confidence-is essential for trustworthy deployment. Existing methods either rely on training-free confidence estimation (e.g., token probabilities, self-consistency) or training-based calibration with correctness annotations. While effective, achieving universal honesty alignment with training-based calibration requires costly, large-scale labeling. To support annotation-efficient training, we introduce Elicitation-Then-Calibration (EliCal), a two-stage framework that first elicits internal confidence using inexpensive self-consistency supervision, then calibrates this confidence with a small set of correctness annotations. To support a large-scale study, we release HonestyBench, a benchmark covering ten free-form QA datasets with 560k training and 70k evaluation instances annotated with correctness and self-consistency signals. Experiments show that EliCal achieves near-optimal alignment with only 1k correctness annotations (0.18% of full supervision) and better alignment performance on unseen MMLU tasks than the calibration-only baseline, offering a scalable solution toward universal honesty alignment in LLMs.

Figures (16)

• Figure 1: The model’s confidence in answering a question is represented by the confidence of its most confident answer, computed via self-consistency as the proportion of generations agreeing with the greedy-search answer (Top). The model’s capability is reflected by the proportion of correct responses, measured as the fraction of generations matching the ground-truth answer (Bottom). These two signals show high correlation across questions.
• Figure 2: Self-consistency confidence vs. correctness on TQ (Qwen2.5-7B-Instruct).
• Figure 3: EliCal reframes honesty alignment as a two-stage learning problem: 1) Confidence Elicitation, which constructs training data from a large set of questions with labels derived through self-consistency; 2) Confidence Calibration, which constructs correctness annotation using a small set of QA pairs to bridge the gap between the model's expressed confidence and its actual accuracy.
• Figure 4: Average performance of training-free methods across all models in the in-domain setting.
• Figure 5: AUROC of EliCal and Cal-Only as the scale of annotated data varies.