Graph-based Confidence Calibration for Large Language Models

TL;DR

The paper addresses the challenge of calibrating LLM confidence by leveraging self-consistency across multiple responses. It proposes a learning-based framework that constructs a consistency graph from candidate answers and uses a graph neural network to estimate the correctness probability for each response, with edge weights based on semantic similarity. The approach demonstrates strong calibration gains (lower ECE, higher AUROC) across multiple QA datasets and shows robust out-of-domain generalization, outperforming several baselines including APRICOT and GraphSpectral variants. This graph-based auxiliary calibration enables more trustworthy LLM deployments by providing reliable confidence estimates and supporting abstention for uncertain cases.

Abstract

Reliable confidence estimation is essential for enhancing the trustworthiness of large language models (LLMs), especially in high-stakes scenarios. Despite its importance, accurately estimating confidence in LLM responses remains a significant challenge. In this work, we propose using an auxiliary learning model to assess response correctness based on the self-consistency of multiple outputs generated by the LLM. Our method builds a consistency graph to represent the agreement among multiple responses and uses a graph neural network (GNN) to estimate the likelihood that each response is correct. Experiments demonstrate that this method has strong calibration performance on various benchmark datasets and generalizes well to out-of-domain cases.

Figures (8)

• Figure 1: The overall framework of our confidence calibration model. Given an input question, our approach first generates multiple responses from the LLM and constructs a similarity-weighted graph based on these responses. This graph serves as the input for the GNN model, which calibrates the confidence of the LLM responses. In the weighted graph, the edge weight $w_{ij}$ is defined as $\mathrm{sim_{cos}}(\mathrm{emb}(r_i), \mathrm{emb}(r_j))$, where $~~ i, j = 1, \ldots, n.$ A higher weight indicates greater similarity between the responses. We also use cluster memberships as the node features to enhance the performance.
• Figure 2: Reliability diagrams for different methods using 10 bins each for Vicuna on TriviaQA. The color, as well as the percentage number within each bar, indicates the proportion of total responses contained in each bin. Larger values are represented by colors closer to purple, and the height indicates the ratio of correct ones. We prefer a wide spread of responses in different bins (strong ability to differentiate responses) and bin heights along the diagonal line (accurate calibration). Our model outperforms others with a broader bin spread and better alignment with the diagonal for calibration accuracy.
• Figure 3: Sensitivity analysis of our model
• Figure 4: Visualization of the generated response patterns
• Figure 5: Evaluation with manual labels (Llama3)