What if Deception Cannot be Detected? A Cross-Linguistic Study on the Limits of Deception Detection from Text
Aswathy Velutharambath, Kai Sassenberg, Roman Klinger
TL;DR
The paper reframes deception from a purely linguistic signal to a belief-based phenomenon, defining deception as misalignment between an author's true beliefs and their expressed arguments. By constructing the DeFaBel corpus suite (German stable-belief, German belief-change, and cross-linguistic English) and conducting cross-dataset, cross-linguistic analyses, the authors show that traditional linguistic cues fail to reliably indicate deception when belief alignment is disentangled from factuality. A comprehensive evaluation of feature-based classifiers, transformer models, and instruction-tuned LLMs reveals near-chance performance on DeFaBel datasets and a pronounced truth-bias in LLMs, suggesting that prior deception-detection successes may stem from dataset artifacts rather than robust cues. The results call for a fundamental rethinking of deception modeling in NLP, urging representations that capture epistemic stance, belief dynamics, and argumentative structure beyond surface text alone, with implications for both research and responsible deployment.
Abstract
Can deception be detected solely from written text? Cues of deceptive communication are inherently subtle, even more so in text-only communication. Yet, prior studies have reported considerable success in automatic deception detection. We hypothesize that such findings are largely driven by artifacts introduced during data collection and do not generalize beyond specific datasets. We revisit this assumption by introducing a belief-based deception framework, which defines deception as a misalignment between an author's claims and true beliefs, irrespective of factual accuracy, allowing deception cues to be studied in isolation. Based on this framework, we construct three corpora, collectively referred to as DeFaBel, including a German-language corpus of deceptive and non-deceptive arguments and a multilingual version in German and English, each collected under varying conditions to account for belief change and enable cross-linguistic analysis. Using these corpora, we evaluate commonly reported linguistic cues of deception. Across all three DeFaBel variants, these cues show negligible, statistically insignificant correlations with deception labels, contrary to prior work that treats such cues as reliable indicators. We further benchmark against other English deception datasets following similar data collection protocols. While some show statistically significant correlations, effect sizes remain low and, critically, the set of predictive cues is inconsistent across datasets. We also evaluate deception detection using feature-based models, pretrained language models, and instruction-tuned large language models. While some models perform well on established deception datasets, they consistently perform near chance on DeFaBel. Our findings challenge the assumption that deception can be reliably inferred from linguistic cues and call for rethinking how deception is studied and modeled in NLP.