Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

When to Trust LLMs: Aligning Confidence with Response Quality

Shuchang Tao, Liuyi Yao, Hanxing Ding, Yuexiang Xie, Qi Cao, Fei Sun, Jinyang Gao, Huawei Shen, Bolin Ding

TL;DR

This work tackles the challenge of trusting LLMs by aligning verbalized confidence with actual answer quality. It introduces CONQORD, a reinforcement learning framework with a dual-component reward: a quality reward $R_Q$ and an order-preserving alignment reward $R_A$, combined as $R_O = R_Q + \alpha R_A$, and optimized via PPO with a KL penalty. Empirical results on TruthfulQA and Natural Questions across multiple foundation models show improved calibration (lower $ECE$) and stronger alignment between confidence and quality, without sacrificing performance, and demonstrate effective use of calibrated confidence to drive adaptive retrieval. The approach advances trustworthy AI by providing transparent, reliable confidence signals that can guide retrieval and decision-making in safety-critical contexts.

Abstract

Despite the success of large language models (LLMs) in natural language generation, much evidence shows that LLMs may produce incorrect or nonsensical text. This limitation highlights the importance of discerning when to trust LLMs, especially in safety-critical domains. Existing methods often express reliability by confidence level, however, their effectiveness is limited by the lack of objective guidance. To address this, we propose CONfidence-Quality-ORDer-preserving alignment approach (CONQORD), which leverages reinforcement learning guided by a tailored dual-component reward function. This function integrates quality reward and order-preserving alignment reward functions. Specifically, the order-preserving reward incentivizes the model to verbalize greater confidence for responses of higher quality to align the order of confidence and quality. Experiments demonstrate that CONQORD significantly improves the alignment performance between confidence and response accuracy, without causing over-cautious. Furthermore, the aligned confidence provided by CONQORD informs when to trust LLMs, and acts as a determinant for initiating the retrieval process of external knowledge. Aligning confidence with response quality ensures more transparent and reliable responses, providing better trustworthiness.

When to Trust LLMs: Aligning Confidence with Response Quality

TL;DR

This work tackles the challenge of trusting LLMs by aligning verbalized confidence with actual answer quality. It introduces CONQORD, a reinforcement learning framework with a dual-component reward: a quality reward and an order-preserving alignment reward , combined as , and optimized via PPO with a KL penalty. Empirical results on TruthfulQA and Natural Questions across multiple foundation models show improved calibration (lower ) and stronger alignment between confidence and quality, without sacrificing performance, and demonstrate effective use of calibrated confidence to drive adaptive retrieval. The approach advances trustworthy AI by providing transparent, reliable confidence signals that can guide retrieval and decision-making in safety-critical contexts.

Abstract

Despite the success of large language models (LLMs) in natural language generation, much evidence shows that LLMs may produce incorrect or nonsensical text. This limitation highlights the importance of discerning when to trust LLMs, especially in safety-critical domains. Existing methods often express reliability by confidence level, however, their effectiveness is limited by the lack of objective guidance. To address this, we propose CONfidence-Quality-ORDer-preserving alignment approach (CONQORD), which leverages reinforcement learning guided by a tailored dual-component reward function. This function integrates quality reward and order-preserving alignment reward functions. Specifically, the order-preserving reward incentivizes the model to verbalize greater confidence for responses of higher quality to align the order of confidence and quality. Experiments demonstrate that CONQORD significantly improves the alignment performance between confidence and response accuracy, without causing over-cautious. Furthermore, the aligned confidence provided by CONQORD informs when to trust LLMs, and acts as a determinant for initiating the retrieval process of external knowledge. Aligning confidence with response quality ensures more transparent and reliable responses, providing better trustworthiness.
Paper Structure (37 sections, 5 equations, 4 figures, 3 tables)

This paper contains 37 sections, 5 equations, 4 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. Confidence Elicitation in LLMs
  4. Confidence Calibration in LLMs
  5. Limitations and Challanges
  6. Confidence Alignment via RL
  7. Preliminary Alignment Approach
  8. Data construction for fine-tuning reward model.
  9. Results and Discussions
  10. CONfidence-Quality-ORDer-preserving Alignment Approach
  11. Dual-component Reward Strategy
  12. Quality reward.
  13. Order-preserving alignment criterion.
  14. Order-preserving alignment reward.
  15. Overall Reward.
  16. ...and 22 more sections

Figures (4)

  • Figure 1: (Top): Vanilla LLMs may generate bad responses and cannot generate confidence. (Middle): Existing methods include verbalizing confidence levels (highlighted in orange) in the output to indicate the model's uncertainty, yet they may still provide bad responses with overly high confidence, revealing a misalignment between expressed confidence and actual response quality. (Bottom): CONQORD aligns with the confidence and response quality.
  • Figure 2: Comparison between vanilla LLAMA-2 7b and PreApproach on TruthfulQA and NQ. Although PreApproach provides better calibration (lower ECE), PreApproach suffers a performance decline.
  • Figure 3: Performance of using calibrated confidence scores as an indicator to determine whether to conduct retrieval augmentation on TruthfulQA and NQ.
  • Figure 4: Impact of coefficient $\alpha$ on confidence alignment and response quality.