Don't Judge a Book by its Cover: Testing LLMs' Robustness Under Logical Obfuscation

TL;DR

This work addresses whether LLMs genuinely reason or merely pattern-match by introducing Logifus, a structure-preserving obfuscation framework, and LogiQAte, a diagnostic benchmark with 1,108 obfuscated questions across four reasoning tasks. The study demonstrates substantial performance degradation under logical obfuscation across multiple models and prompting strategies, highlighting brittleness in current reasoning capabilities. Through memorization and layer-wise confidence analyses, the authors show that obfuscated prompts induce greater reliance on training data and weaker deep-layer consolidation, suggesting surface-form reliance rather than true comprehension. The findings underscore the need for models that maintain semantic understanding under surface-form transformations, with implications for robust evaluation and the development of deeper reasoning in LLMs.

Abstract

Tasks such as solving arithmetic equations, evaluating truth tables, and completing syllogisms are handled well by large language models (LLMs) in their standard form, but they often fail when the same problems are posed in logically equivalent yet obfuscated formats. To study this vulnerability, we introduce Logifus, a structure-preserving logical obfuscation framework, and, utilizing this, we present LogiQAte, a first-of-its-kind diagnostic benchmark with 1,108 questions across four reasoning tasks: (i) Obfus FOL (first-order logic entailment under equivalence-preserving rewrites), (ii) Obfus Blood Relation (family-graph entailment under indirect relational chains), (iii) Obfus Number Series (pattern induction under symbolic substitutions), and (iv) Obfus Direction Sense (navigation reasoning under altered directions and reference frames). Across all the tasks, evaluating six state-of-the-art models, we find that obfuscation severely degrades zero-shot performance, with performance dropping on average by 47% for GPT-4o, 27% for GPT-5, and 22% for reasoning model, o4-mini. Our findings reveal that current LLMs parse questions without deep understanding, highlighting the urgency of building models that genuinely comprehend and preserve meaning beyond surface form.

Figures (6)

• Figure 1: An Overview of LLMs under Logical Obfuscation with Logifus. On the left, an obfuscated version of a question, while remaining logically equivalent, poses significant challenge to the model. On the right, we track the model’s confidence in predicting the correct next word or token, quantified as the layer-wise next-token log probability assigned to that token at each step. Under logical obfuscation, these layer-wise scores drop sharply as the question is processed, indicating a transition into a reasoning-failure zone (cf. Section \ref{['layerwise']}). The plot shows this degradation across the first 15 layers of LLaMA 3.1 8B touvron2023llama.
• Figure 2: Logical obfuscation degrades performance across tasks in LogiQAte. We report relative accuracy for obfuscated questions against base questions (performance fixed at 1.00) across five LLMs under zero-shot settings. Across all tasks and settings, we observe consistent degradation, up to $-86\%$, highlighting the brittleness of LLM reasoning under logical obfuscation.
• Figure 3: Overview of LogiQAte and Logifus. From a user question (left), four task types are generated: Obfus FOL, Obfus Blood Relation, Obfus Number Series, and Obfus Direction Sense. Each task applies a logical equivalence-preserving transformation and undergoes human (along with Prover9 prover9-mace4 in the case of Obfus FOL) verification. The right panel summarizes the dataset statistics: 1,108 questions across four tasks.
• Figure 4: Obfuscation reduces performance across types in Obfus Number Series. Accuracy drops range from 20-45% for reasoning-focused models and up to 80-90% for general-purpose models such as Claude 3.7 Sonnet and Gemini 2.5 Pro, which already struggle on base questions. GPT-5 maintains higher accuracy across all obfuscation and base variants.
• Figure 5: Obfuscation increases prediction AUROC (indicating greater memorization) by 12-15% in Obfus FOL and 20-25% in Obfus Blood Relation across both L1 and L2 levels (L2 being higher). The y-axis shows Prediction AUROC (%), and the x-axis denotes K, the fraction of least-likely tokens used in computing membership scores. Plotting Prediction AUROC against K illustrates how obfuscation amplifies memorization cues, making models rely more on training data patterns than genuine reasoning.