Coherent Without Grounding, Grounded Without Success: Observability and Epistemic Failure

Abstract

When an agent can articulate why something works, we typically take this as evidence of genuine understanding. This presupposes that effective action and correct explanation covary, and that coherent explanation reliably signals both. I argue that this assumption fails for contemporary Large Language Models (LLMs). I introduce what I call the Bidirectional Coherence Paradox: competence and grounding not only dissociate but invert across epistemic conditions. In low-observability domains, LLMs often act successfully while misidentifying the mechanisms that produce their success. In high-observability domains, they frequently generate explanations that accurately track observable causal structure yet fail to translate those diagnoses into effective intervention. In both cases, explanatory coherence remains intact, obscuring the underlying dissociation. Drawing on experiments in compiler optimization and hyperparameter tuning, I develop the Epistemic Triangle, a model of how priors, signals, and domain knowledge interact under varying observability. The results suggest that neither behavioral success nor explanatory accuracy alone suffices for attributing understanding. I argue that evaluating artificial epistemic agents requires a tripartite framework -- coherence, grounding, and a proper basing relation linking explanation to action. The systematic separation of knowing-that and knowing-how in LLMs thus challenges assumptions inherited from both epistemology and current AI evaluation practice.

Figures (6)

• Figure 1: The Bidirectional Coherence Paradox: Case Studies. Left: In low-observability domains (compiler optimization), actions succeed despite ungrounded explanations---the LLM achieves performance gains through mechanisms it misidentifies. Right: In high-observability domains (HPO), explanations are grounded but actions sometimes fail---correct diagnosis does not guarantee effective intervention.
• Figure 2: The Bidirectional Coherence Paradox. Direct comparison of ActSR and ASR across observability regimes. Type A (left): actions succeed while explanations fail. Type B (right): explanations succeed while actions fail. The 61-point swing demonstrates that coherence and grounding can be inversely related.
• Figure 3: The Epistemic Triangle. LLM reasoning quality emerges from the interaction of three information sources: weak observational signals, strong priors from training, and domain knowledge providing semantic scaffolding. The model explains how observability determines which vertex dominates, producing different failure modes.
• Figure 4: The Observability Spectrum and Paradox Inversion. The Epistemic Triangle predicts that observability determines which vertex dominates reasoning, producing qualitatively different failure modes. Low observability yields prior-dominated reasoning where actions succeed but explanations fail (Type A). High observability yields signal-dominated reasoning where explanations succeed but actions fail (Type B).
• Figure 5: Overview of the MRL pipeline in the first domain: compiler optimization