Improving Chain-of-Thought for Logical Reasoning via Attention-Aware Intervention

TL;DR

This work tackles non-interactive, end-to-end logical reasoning with LLMs by embedding symbolic structure directly into prompts and guiding the model’s inference through attention-level interventions. It first analyzes how Symbolic-Aided CoT prompts induce specialized attention-head patterns, identifying anchor, aggregation, and copy heads aligned with reasoning operators. It then introduces Attention-Aware Intervention (AAI), a lightweight, model-agnostic masking-based reweighting mechanism that grounds rule references in their semantic content via tailored masks and head selection, incurring negligible computational overhead. Across multiple benchmarks (ProofWriter, PrOntoQA, Logical Deduction, FOLIO, GSM8k) and model scales (1.7B–32B), AAI yields consistent accuracy gains and demonstrates robustness to hyperparameters, with greater gains as model size increases and broad compatibility with different prompting strategies. These results advance scalable, interpretable reasoning in LLMs by enabling internal symbolic reasoning without external solvers or multi-step interactions, offering practical impact for robust reasoning in real-world tasks.

Abstract

Modern logical reasoning with LLMs primarily relies on employing complex interactive frameworks that decompose the reasoning process into subtasks solved through carefully designed prompts or requiring external resources (e.g., symbolic solvers) to exploit their strong logical structures. While interactive approaches introduce additional overhead, hybrid approaches depend on external components, which limit their scalability. A non-interactive, end-to-end framework enables reasoning to emerge within the model itself -- improving generalization while preserving analyzability without any external resources. In this work, we introduce a non-interactive, end-to-end framework for reasoning tasks. We show that introducing structural information into the few-shot prompt activates a subset of attention heads that patterns aligned with logical reasoning operators. Building on this insight, we propose Attention-Aware Intervention (AAI), an inference-time intervention method that reweights attention scores across selected heads identified by their logical patterns. AAI offers an efficient way to steer the model's reasoning toward leveraging prior knowledge through attention modulation. Extensive experiments show that AAI enhances logical reasoning performance across diverse benchmarks and model architectures, while incurring negligible additional computational overhead. Code is available at https://github.com/phuongnm94/aai_for_logical_reasoning.

Figures (8)

• Figure 1: Overview of the comparison between the proposed method and existing approaches.
• Figure 2: Visualization of attention heads in the Qwen3-8B model on an example from the ProofWriter dataset. Higher attention scores are represented by stronger colors. For ease of interpretation, every five subword tokens are grouped along both the x- and y-axes in the figure caption. A more detailed (or full-scale) version of this figure is provided in Fig. \ref{['fig_attention_prelim_exp_full']}).
• Figure 3: Ablation study on our AAI method.
• Figure 4: Performance across different model sizes of Qwen-3 with three prompting techniques on three datasets.
• Figure 5: Performance (accuracy) changes with respect to different hyperparameter values (attention binarization threshold, $s^{diagonal}$, reweighting coefficient ($c$) and bias constant ($b$) in Eq. \ref{['eq_reweighting']}) on two benchmark datasets, Logical Deduction and ProofWriter. The dashed line denotes the baseline system, which does not use our AAI mechanism. Values marked with underscores indicate the default settings used in Table \ref{['tab_mainresult']} ($binar.=0.04$, $diagon.r.=0.3$, $coef.=1.0$, $bias=0$). In the last column, the coefficient is set to $coef.=0$ and the bias value is varied.