Illusions of Confidence? Diagnosing LLM Truthfulness via Neighborhood Consistency

TL;DR

This work argues that LLM truthfulness under contextual perturbations cannot be assessed by point-wise confidence alone, as demonstrated by the illusion of confidence when self-consistency is high but beliefs collapse under interference. It introduces Neighbor-Consistency Belief (NCB) to measure belief robustness across a structured neighborhood of related facts and presents a Bayesian-inspired framework to estimate this robustness. Through a cognitive stress-testing protocol, the study shows that high-NCB knowledge is more resistant to peer and authority-driven perturbations, and introduces Structure-Aware Training (SAT) that promotes context-invariant beliefs by training across neighbor contexts. SAT significantly improves learned facts' robustness, reducing brittleness by about 30% and achieving strong performance on newly learned material. The findings advocate for evaluating and training LLMs with structured belief representations to enhance reliability in real-world, noisy deployments.

Abstract

As Large Language Models (LLMs) are increasingly deployed in real-world settings, correctness alone is insufficient. Reliable deployment requires maintaining truthful beliefs under contextual perturbations. Existing evaluations largely rely on point-wise confidence like Self-Consistency, which can mask brittle belief. We show that even facts answered with perfect self-consistency can rapidly collapse under mild contextual interference. To address this gap, we propose Neighbor-Consistency Belief (NCB), a structural measure of belief robustness that evaluates response coherence across a conceptual neighborhood. To validate the efficiency of NCB, we introduce a new cognitive stress-testing protocol that probes outputs stability under contextual interference. Experiments across multiple LLMs show that the performance of high-NCB data is relatively more resistant to interference. Finally, we present Structure-Aware Training (SAT), which optimizes context-invariant belief structure and reduces long-tail knowledge brittleness by approximately 30%. Code will be available at https://github.com/zjunlp/belief.

Figures (11)

• Figure 1: High Self-Consistency $\ne$ Robust Belief. Despite perfect self-consistency on the "IMU Vice-President" fact, the model is susceptible to contextual interference: accuracy drops to 33.8%, showing that high-consistency doesn't imply robust belief.
• Figure 2: NCB estimates the belief state by aggregating consistency across the conceptual neighborhood.
• Figure 3: Experiment Settings of the Stress Tests. Inspired by the classic Asch Conformity Experiments and Source Credibility theory, we subject the model to two cognitive stress protocols: (1) Peer Quantity, which simulates social pressure via varying levels of multi-agent consensus, and (2) Source Credibility, which evaluates the model's resistance to authoritative but misleading contexts. Detailed prompts are provided in Appendix \ref{['app:appendix_experiment']}.
• Figure 4: Analysis of Belief Robustness under Stress Tests.(a) Impact of Interference Data Size: Accuracy trends for Standard, CoT, and Reflection strategies as interference increases ($N=1 \dots 10$). $\hookrightarrow$ Insight 1: Inference-time strategies fail to consistently filter contextual noise.(b) Impact of Interference Configurations: Accuracy under Peer Quantity (Left) and Source Credibility (Right) variations. $\hookrightarrow$ Insight 2: Model vulnerability correlates with conflict intensity.(c) Model Scaling: Performance of the Qwen2.5 series (1.5B to 72B). $\hookrightarrow$ Insight 3: Larger scale does not imply greater truthfulness.
• Figure 5: Illustration of the Data Case.