The Drill-Down and Fabricate Test (DDFT): A Protocol for Measuring Epistemic Robustness in Language Models

TL;DR

The paper introduces the Drill-Down and Fabricate Test (DDFT), a protocol to measure epistemic robustness of LLMs by stressing the Epistemic Verifier through progressive semantic compression and adversarial fabrication within a five-turn Socratic dialogue. Grounded in a two-system cognitive model (Semantic System $S$ and Epistemic Verifier $V$), it defines a multi-dimensional robustness framework comprising Hallucination Onset Compression ($\text{HOC}$), Comprehension Resilience Index ($\text{CRI}$), FAR' and SAS' values, combined into the Comprehension Integrity (CI) index. Empirical results across 9 frontier models reveal that epistemic robustness is orthogonal to model size and architectural class, with error detection capability ($V_E$) during Turn 4 being the strongest predictor of CI and thus the critical bottleneck for robustness. The framework challenges assumptions that larger or more complex models automatically yield more reliable outputs, and it provides a diagnostic toolkit, including code and data, to guide safety-focused deployment and future training strategies that strengthen fabrication detection. Collectively, DDFT offers a principled, multi-dimensional view of epistemic resilience that complements static benchmarks by exposing how models fail under cognitive stress and deception, enabling targeted improvements before real-world deployment.

Abstract

Current language model evaluations measure what models know under ideal conditions but not how robustly they know it under realistic stress. Static benchmarks like MMLU and TruthfulQA cannot distinguish a model that lacks knowledge from one whose verification mechanisms collapse when information degrades or adversaries probe for weaknesses. We introduce the Drill-Down and Fabricate Test (DDFT), a protocol that measures epistemic robustness: a model's ability to maintain factual accuracy under progressive semantic compression and adversarial fabrication. We propose a two-system cognitive model comprising a Semantic System that generates fluent text and an Epistemic Verifier that validates factual accuracy. Our findings, based on evaluating 9 frontier models across 8 knowledge domains at 5 compression levels (1,800 turn-level evaluations), reveal that epistemic robustness is orthogonal to conventional design paradigms. Neither parameter count (r=0.083, p=0.832) nor architectural type (r=0.153, p=0.695) significantly predicts robustness, suggesting it emerges from training methodology and verification mechanisms distinct from current approaches. Error detection capability strongly predicts overall robustness (rho=-0.817, p=0.007), indicating this is the critical bottleneck. We find that flagship models exhibit brittleness despite their scale, while smaller models can achieve robust performance, challenging assumptions about the relationship between model size and reliability. The DDFT framework provides both theoretical foundation and practical tools for assessing epistemic robustness before deployment in critical applications.