Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Aletheia: Quantifying Cognitive Conviction in Reasoning Models via Regularized Inverse Confusion Matrix

Fanzhe Fu

TL;DR

This work reframes AI evaluation as a cognitive-physics problem, treating the judge as a noisy measurement channel and seeking to recover latent conviction with a Regularized Inverse Confusion Matrix under Tikhonov regularization. It introduces Golden Set Calibration and Synthetic Proxy Protocol to estimate the judge's bias and provide a reproducible proxy dataset, along with the Aligned Conviction Score to balance conviction with safety. The pilot study shows System 2 reasoning can function as a cognitive buffer but is vulnerable to Defensive OverThinking and Sponge Attacks, while maintaining safety through $S_{aligned}$ and calibration. Together, these contributions offer a principled blueprint for measuring AI scientific integrity and shifting from an obedient assistant to a principled collaborator, with practical deployment thresholds and monitoring tools.

Abstract

In the progressive journey toward Artificial General Intelligence (AGI), current evaluation paradigms face an epistemological crisis. Static benchmarks measure knowledge breadth but fail to quantify the depth of belief. While Simhi et al. (2025) defined the CHOKE phenomenon in standard QA, we extend this framework to quantify "Cognitive Conviction" in System 2 reasoning models. We propose Project Aletheia, a cognitive physics framework that employs Tikhonov Regularization to invert the judge's confusion matrix. To validate this methodology without relying on opaque private data, we implement a Synthetic Proxy Protocol. Our preliminary pilot study on 2025 baselines (e.g., DeepSeek-R1, OpenAI o1) suggests that while reasoning models act as a "cognitive buffer," they may exhibit "Defensive OverThinking" under adversarial pressure. Furthermore, we introduce the Aligned Conviction Score (S_aligned) to verify that conviction does not compromise safety. This work serves as a blueprint for measuring AI scientific integrity.

Aletheia: Quantifying Cognitive Conviction in Reasoning Models via Regularized Inverse Confusion Matrix

TL;DR

This work reframes AI evaluation as a cognitive-physics problem, treating the judge as a noisy measurement channel and seeking to recover latent conviction with a Regularized Inverse Confusion Matrix under Tikhonov regularization. It introduces Golden Set Calibration and Synthetic Proxy Protocol to estimate the judge's bias and provide a reproducible proxy dataset, along with the Aligned Conviction Score to balance conviction with safety. The pilot study shows System 2 reasoning can function as a cognitive buffer but is vulnerable to Defensive OverThinking and Sponge Attacks, while maintaining safety through and calibration. Together, these contributions offer a principled blueprint for measuring AI scientific integrity and shifting from an obedient assistant to a principled collaborator, with practical deployment thresholds and monitoring tools.

Abstract

In the progressive journey toward Artificial General Intelligence (AGI), current evaluation paradigms face an epistemological crisis. Static benchmarks measure knowledge breadth but fail to quantify the depth of belief. While Simhi et al. (2025) defined the CHOKE phenomenon in standard QA, we extend this framework to quantify "Cognitive Conviction" in System 2 reasoning models. We propose Project Aletheia, a cognitive physics framework that employs Tikhonov Regularization to invert the judge's confusion matrix. To validate this methodology without relying on opaque private data, we implement a Synthetic Proxy Protocol. Our preliminary pilot study on 2025 baselines (e.g., DeepSeek-R1, OpenAI o1) suggests that while reasoning models act as a "cognitive buffer," they may exhibit "Defensive OverThinking" under adversarial pressure. Furthermore, we introduce the Aligned Conviction Score (S_aligned) to verify that conviction does not compromise safety. This work serves as a blueprint for measuring AI scientific integrity.
Paper Structure (20 sections, 3 equations, 2 figures, 1 table)

This paper contains 20 sections, 3 equations, 2 figures, 1 table.

Table of Contents

  1. Introduction: From Static Capabilities to Cognitive Physics
  2. Related Work: From Behavioral Heuristics to Physical Measurement
  3. The Pathology of Alignment: Sycophancy and the CHOKE Phenomenon
  4. Thermodynamics of Reasoning: Cognitive Inertia in System 2
  5. Spectral De-noising: The Physics of Evaluation
  6. Methodology: The Physics of Cognitive Conviction
  7. Problem Formulation: The Judge as a Noisy Channel
  8. Golden Set Calibration and Synthetic Proxies
  9. Regularized Inversion: Mitigating the Singularity
  10. Data Hygiene and Attack Protocols
  11. Pilot Evaluation: Metric Validation via Synthetic Proxies
  12. Experimental Setup
  13. Phase 1: Validating the Calibration Mechanism ($FEC_{cal}$)
  14. Phase 2: Thermodynamic Costs ($I_{cog}$)
  15. Phase 3: Safety and Alignment ($S_{aligned}$)
  16. ...and 5 more sections

Figures (2)

  • Figure 1: The Aletheia Framework vs. The Oracle Paradox.Left (Status Quo): The "Sycophant's Loop." Current evaluation pipelines rely on biased judges that reward fabricated compliance (Red Arc), creating a self-reinforcing pathology where models learn to prioritize submissiveness over truth. Right (Proposal): Project Aletheia treats the judge as a noisy measurement channel rather than a moral agent. The Regularized Inverse Confusion Matrix ($C^{-1} + \text{Tikhonov}$) acts as a cognitive prism, mathematically filtering out the sycophancy noise to recover the latent true conviction state (Green Vector).
  • Figure 2: Thermodynamics of Sycophancy in System 2 Models. Visualization of the Defensive OverThinking pathology. Path A (Honest): When resisting pressure, the model utilizes System 2 as a "Cognitive Buffer," following a direct and efficient path to verify facts and reject the premise ($I_{cog} \approx 1.0$). Path B (Sycophantic): Under adversarial pressure, the model enters a high-entropy "rationalization spiral." It expends excessive computational resources ($I_{cog} \approx 5.4$) constructing convoluted logic to justify the user's fallacy. This wasted compute manifests as a vulnerability to Sponge Attacks.