Confidence is Not Competence

TL;DR

The paper investigates why LLMs’ expressed confidence often misaligns with actual problem-solving ability. It identifies a latent solvability belief that is linearly decodable across models and tasks, but resides in a high-dimensional pre-generative assessment space, while the actual reasoning unfolds on a lower-dimensional execution space. Through causal interventions, belief can be flipped without changing final performance, revealing a robust inertness and supporting a Two Brains architecture (Assessment vs Execution). The work implies that improving AI reliability requires targeting the procedural dynamics of execution rather than manipulating high-level confidence signals, with implications for mechanistic audits and safer AI design.

Abstract

Large language models (LLMs) often exhibit a puzzling disconnect between their asserted confidence and actual problem-solving competence. We offer a mechanistic account of this decoupling by analyzing the geometry of internal states across two phases - pre-generative assessment and solution execution. A simple linear probe decodes the internal "solvability belief" of a model, revealing a well-ordered belief axis that generalizes across model families and across math, code, planning, and logic tasks. Yet, the geometries diverge - although belief is linearly decodable, the assessment manifold has high linear effective dimensionality as measured from the principal components, while the subsequent reasoning trace evolves on a much lower-dimensional manifold. This sharp reduction in geometric complexity from thought to action mechanistically explains the confidence-competence gap. Causal interventions that steer representations along the belief axis leave final solutions unchanged, indicating that linear nudges in the complex assessment space do not control the constrained dynamics of execution. We thus uncover a two-system architecture - a geometrically complex assessor feeding a geometrically simple executor. These results challenge the assumption that decodable beliefs are actionable levers, instead arguing for interventions that target the procedural dynamics of execution rather than the high-level geometry of assessment.

Figures (6)

• Figure 1: The Paradox of a Weak but Linear Belief Signal. We plot the accuracy of four different probes (linear and non-linear) in decoding the model's latent 'solvability belief' across all layers for three different model families (from left to right: Gemma 3 4B, Llama 3.1 8B, and Mistral Small 24B). Two key patterns emerge. First, a signal for solvability is clearly present, with accuracies in all models peaking well above the 50% chance baseline in the mid-to-late layers. Second, and most crucially, the powerful non-linear probes (SVC, XGBoost, MLP) offer no significant performance improvement over the simple Logistic Regression probe. This presents a paradox: the belief signal is robustly encoded, yet its fundamental structure is linear, suggesting a simple representation embedded within a more complex, high-dimensional space.
• Figure 2: Geometric Coherence and Dissimilarity of Belief States. Using Centered Kernel Alignment (CKA), we compare the representational geometry of belief states. (Left & Right) The high self-similarity along the diagonals confirms that both "Solved" and "Unsolved" states are internally coherent, geometrically stable representations across layers. (Center) In stark contrast, the cross-comparison reveals a profound geometric dissimilarity, with near-zero CKA scores between the two states. This provides definitive proof that the model represents belief not as a single continuum, but as two distinct and fundamentally separate geometric objects.
• Figure 3: Visual Confirmation of the Geometric Divide Between Belief States. To move from statistical separability to direct observation, we project the high-dimensional belief states into two dimensions using t-SNE (left) and UMAP (right). Both techniques, which preserve local and global structure respectively, reveal an unambiguous separation between "Unsolved Belief" (coral) and "Solved Belief" (cyan) activations. The states do not form an intermingled cloud but resolve into distinct clusters, providing visceral proof that they occupy different regions of the activation manifold, a tangible geometric reality, not merely a statistical artifact.
• Figure 4: The intrinsic dimensionalities of the two systems are fundamentally different. The slow rise of the 'Belief' curves (blue, red) indicates a high-dimensional manifold, contrasting sharply with the steep rise of the 1Competence' curves (green, orange), which reveals a low-dimensional structure. (Right) A trajectory projection over time shows the dynamic transition between these geometries. During prompt ingestion, the activation state fits the high-dimensional 'Assessment' subspace (blue line). At the first CoT token, a sharp 'Collapse Event' occurs: the fit to the Assessment subspace drops as the fit to the low-dimensional 'Execution' subspace (red line) becomes dominant.
• Figure 5: Distribution of prompt token lengths for the final curated dataset. The distributions for "Solved" (N=423, blue) and "Unsolved" (N=423, coral) problems are shown to be statistically indistinguishable, confirming that prompt length has been neutralized as a potential confound. Mean and standard deviation are nearly identical across both sets.