Abstract Activation Spaces for Content-Invariant Reasoning in Large Language Models

TL;DR

This work tackles the content effect in LLM syllogistic reasoning by introducing abstraction-guided activation steering. It constructs an abstract reasoning space from paired content-laden and abstract syllogisms and learns lightweight Abstractors to map content-conditional activations into this space, integrating the predictions during inference without weight updates. Through cross-lingual evaluation, the method achieves strong, zero-shot transfer in high-resource languages and meaningful gains in low-resource languages, evidenced by substantial increases in Bias-Penalized Accuracy and favorable Abstract Alignment. The results demonstrate that activation-level abstraction can robustly dissociate structural inference from semantic content, offering a scalable, modular approach to improve formal reasoning in LLMs while preserving general language abilities. Limitations include the focus on syllogistic reasoning and open-weight models, with future work needed to extend to broader reasoning tasks and closed systems.

Abstract

Large Language Models (LLMs) often struggle with deductive judgment in syllogistic reasoning, systematically conflating semantic plausibility with formal validity a phenomenon known as content effect. This bias persists even when models generate step-wise explanations, indicating that intermediate rationales may inherit the same semantic shortcuts that affect answers. Recent approaches propose mitigating this issue by increasing inference-time structural constraints, either by encouraging abstract intermediate representations or by intervening directly in the model's internal computations; however, reliably suppressing semantic interference remains an open challenge. To make formal deduction less sensitive to semantic content, we introduce a framework for abstraction-guided reasoning that explicitly separates structural inference from lexical semantics. We construct paired content-laden and abstract syllogisms and use the model's activations on abstract inputs to define an abstract reasoning space. We then learn lightweight Abstractors that, from content-conditioned residual-stream states, predict representations aligned with this space and integrate these predictions via multi-layer interventions during the forward pass. Using cross-lingual transfer as a test bed, we show that abstraction-aligned steering reduces content-driven errors and improves validity-sensitive performance. Our results position activation-level abstraction as a scalable mechanism for enhancing the robustness of formal reasoning in LLMs against semantic interference.

Figures (15)

• Figure 1: Overview of the abstraction steering framework. (i) The model processes both content-laden ("All flowers need water ...") and abstract ("All X need Y ...") syllogisms. Abstractors learn to map content-laden activations into the abstract reasoning space, (ii) integrating predicted targets via multi-layer activation at inference-time.
• Figure 2: Performance on the English test set. Global Accuracy (blue) and Bias-Penalised Accuracy (green); darker shades indicate gains from activation steering.
• Figure 3: Performance comparison on HRLs (French, Spanish, Italian, German, Russian, Chinese).
• Figure 4: Performance comparison on LRLs (Bengali, Swahili, Telugu).
• Figure 5: Performance impact on MMLU. Factual tasks (orange) and formal logic subtasks (green).