Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Double-Calibration: Towards Trustworthy LLMs via Calibrating Knowledge and Reasoning Confidence

Yuyin Lu, Ziran Liang, Yanghui Rao, Wenqi Fan, Fu Lee Wang, Qing Li

TL;DR

This work tackles the hallucination and uncertainty challenges in large language models by introducing DoublyCal, a doubly calibrated KG‑RAG framework. It first calibrates KG evidence confidence via a Bayesian Beta‑Bernoulli model over constrained relational paths, then trains a lightweight reasoning proxy with SFT and RL to generate calibrated evidence for an open LLM. The LLM reasons over this externally grounded, calibrated evidence to produce final answers with traceable confidence, achieving improved accuracy and ECE on knowledge‑intensive KGQA benchmarks with low token cost. The approach demonstrates significant gains in reliability and trustworthiness, and shows cross‑model compatibility and efficiency advantages for real‑world knowledge tasks.

Abstract

Trustworthy reasoning in Large Language Models (LLMs) is challenged by their propensity for hallucination. While augmenting LLMs with Knowledge Graphs (KGs) improves factual accuracy, existing KG-augmented methods fail to quantify epistemic uncertainty in both the retrieved evidence and LLMs' reasoning. To bridge this gap, we introduce DoublyCal, a framework built on a novel double-calibration principle. DoublyCal employs a lightweight proxy model to first generate KG evidence alongside a calibrated evidence confidence. This calibrated supporting evidence then guides a black-box LLM, yielding final predictions that are not only more accurate but also well-calibrated, with confidence scores traceable to the uncertainty of the supporting evidence. Experiments on knowledge-intensive benchmarks show that DoublyCal significantly improves both the accuracy and confidence calibration of black-box LLMs with low token cost.

Double-Calibration: Towards Trustworthy LLMs via Calibrating Knowledge and Reasoning Confidence

TL;DR

This work tackles the hallucination and uncertainty challenges in large language models by introducing DoublyCal, a doubly calibrated KG‑RAG framework. It first calibrates KG evidence confidence via a Bayesian Beta‑Bernoulli model over constrained relational paths, then trains a lightweight reasoning proxy with SFT and RL to generate calibrated evidence for an open LLM. The LLM reasons over this externally grounded, calibrated evidence to produce final answers with traceable confidence, achieving improved accuracy and ECE on knowledge‑intensive KGQA benchmarks with low token cost. The approach demonstrates significant gains in reliability and trustworthiness, and shows cross‑model compatibility and efficiency advantages for real‑world knowledge tasks.

Abstract

Trustworthy reasoning in Large Language Models (LLMs) is challenged by their propensity for hallucination. While augmenting LLMs with Knowledge Graphs (KGs) improves factual accuracy, existing KG-augmented methods fail to quantify epistemic uncertainty in both the retrieved evidence and LLMs' reasoning. To bridge this gap, we introduce DoublyCal, a framework built on a novel double-calibration principle. DoublyCal employs a lightweight proxy model to first generate KG evidence alongside a calibrated evidence confidence. This calibrated supporting evidence then guides a black-box LLM, yielding final predictions that are not only more accurate but also well-calibrated, with confidence scores traceable to the uncertainty of the supporting evidence. Experiments on knowledge-intensive benchmarks show that DoublyCal significantly improves both the accuracy and confidence calibration of black-box LLMs with low token cost.
Paper Structure (60 sections, 7 equations, 4 figures, 6 tables)

This paper contains 60 sections, 7 equations, 4 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Knowledge-Augmented Generation for Trustworthy LLMs
  4. Uncertainty Quantification for LLMs
  5. Preliminaries
  6. Uncertainty and Confidence
  7. Knowledge Graph
  8. Knowledge Graph Question Answering
  9. Methodology
  10. Bayesian Calibration of KG Evidence
  11. KG Evidence Formulation.
  12. Example.
  13. Confidence Estimation with Beta-Bernoulli Model.
  14. Example (cont.).
  15. Reasoning Proxy for Evidence Generation and Calibration
  16. ...and 45 more sections

Figures (4)

  • Figure 1: A motivating example of double-calibration against hallucination in KG-augmented LLMs.
  • Figure 2: The DoublyCal framework: (Top-right) Bayesian calibration of Knowledge Graph (KG) evidence confidence; (Top-left) Supervised fine-tuning and reinforcement learning of a proxy evidence generator; (Bottom) Evidence-guided inference where the proxy’s outputs elicit a black-box LLM to produce final answers with well-calibrated confidence.
  • Figure 3: Token efficiency analysis on WebQSP.
  • Figure 4: Compatibility analysis of DoublyCal across diverse black-box LLMs. (a) F1 and ECE with/without DoublyCal. (b–e) Calibration diagrams (bars: confidence distribution per confidence bin, line: empirical accuracy, dashed: ideal calibration).