Influential Training Data Retrieval for Explaining Verbalized Confidence of LLMs

TL;DR

This work addresses why LLMs verbalize confidence and whether that confidence is grounded in content rather than superficial cues. It introduces TracVC, a data-centric framework that combines information retrieval and gradient-based influence estimation to trace verbalized confidence back to training data, and defines content groundness via the $ccr$ metric to quantify grounding in content-related examples versus confidence cues. Across 11 open-source LLMs (including OLMo variants and Llama3) and five QA benchmarks, the study finds that larger models are not consistently more content-grounded, and post-training schemes can shift grounding in either direction; importantly, confidence-related data can dominate for some models, indicating a misalignment between sounding confident and warranted confidence. The results offer a scalable, data-driven lens for explaining LLM behavior and lay groundwork for training strategies aimed at more trustworthy and content-grounded expressions of confidence.

Abstract

Large language models (LLMs) can increase users' perceived trust by verbalizing confidence in their outputs. However, prior work has shown that LLMs are often overconfident, making their stated confidence unreliable since it does not consistently align with factual accuracy. To better understand the sources of this verbalized confidence, we introduce TracVC (\textbf{Trac}ing \textbf{V}erbalized \textbf{C}onfidence), a method that builds on information retrieval and influence estimation to trace generated confidence expressions back to the training data. We evaluate TracVC on OLMo and Llama models in a question answering setting, proposing a new metric, content groundness, which measures the extent to which an LLM grounds its confidence in content-related training examples (relevant to the question and answer) versus in generic examples of confidence verbalization. Our analysis reveals that OLMo2-13B is frequently influenced by confidence-related data that is lexically unrelated to the query, suggesting that it may mimic superficial linguistic expressions of certainty rather than rely on genuine content grounding. These findings point to a fundamental limitation in current training regimes: LLMs may learn how to sound confident without learning when confidence is justified. Our analysis provides a foundation for improving LLMs' trustworthiness in expressing more reliable confidence.

Figures (7)

• Figure 1: The workflow of TracVC. We first retrieve the top 10 relevant samples for content (question ($q$) and answer ($a$)) and confidence (prompt ($p$) and confidence ($c$)) segment in the test sample. Then, we compute and compare the influence score regarding the confidence generation for content-related and confidence-related data samples.
• Figure 2: Examples of the most influential training samples for different LLMs when generating confidence. Retrieved samples come from pre-training (Source: Pre) and post-training (Source: Post) corpora. Ground-truth answers are shown in bold.
• Figure 3: Proportion of sources of the most influential training examples for each test sample. The most influential example is defined as the one with the highest influence score among all retrieved examples from both pre-training and post-training corpora. For example, Pre-Content denotes a content-related example from the pre-training corpus.
• Figure 4: Distribution of mean and standard deviation of influence scores across the 10 retrieved training examples for each test sample. Each point presents a test sample.
• Figure 5: Pearson correlation between content groundness and task accuracy. Each data point reflects the accuracy of the dataset and the ccr score evaluated with this dataset. Each dataset is evaluated with three settings (pre, post and pre+post), thus these three settings can have different ccr scores but the same accuracy.