Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Scaling Reasoning without Attention

Xueliang Zhao, Wei Wu, Lingpeng Kong

TL;DR

This work tackles the inefficiency of attention-based Transformers in long-context, high-difficulty reasoning tasks by introducing PromptCoT-Mamba, an attention-free model built on state-space dual (SSD) layers from Mamba-2. It achieves constant-time, fixed-memory inference per token, avoiding key-value caching, while attaining competitive and, on several benchmarks, state-of-the-art reasoning performance through a two-phase curriculum fine-tuning regime based on the PromptCoT synthesis paradigm. Key contributions include the SSD-based architecture with $O(NP)$ per-step inference, a scalable $O(TNP)$ training formulation via structured contractions, and a principled data-centric training pipeline that combines foundational reasoning data with rationale-guided PromptCoT examples. The results show PromptCoT-Mamba-7B outperforming comparable Transformer and hybrid baselines and surpassing larger models on tasks like AIME and LiveCodeBench, indicating that state-space models can offer efficient, scalable alternatives for complex reasoning in resource-constrained settings.

Abstract

Large language models (LLMs) have made significant advances in complex reasoning tasks, yet they remain bottlenecked by two core challenges: architectural inefficiency due to reliance on Transformers, and a lack of structured fine-tuning for high-difficulty domains. We introduce \ourmodel, an attention-free language model that addresses both issues through architectural and data-centric innovations. Built on the state space dual (SSD) layers of Mamba-2, our model eliminates the need for self-attention and key-value caching, enabling fixed-memory, constant-time inference. To train it for complex reasoning, we propose a two-phase curriculum fine-tuning strategy based on the \textsc{PromptCoT} synthesis paradigm, which generates pedagogically structured problems via abstract concept selection and rationale-guided generation. On benchmark evaluations, \ourmodel-7B outperforms strong Transformer and hybrid models of comparable scale, and even surpasses the much larger Gemma3-27B by 2.6\% on AIME 24, 0.6\% on AIME 25, and 3.0\% on Livecodebench. These results highlight the potential of state space models as efficient and scalable alternatives to attention-based architectures for high-capacity reasoning.

Scaling Reasoning without Attention

TL;DR

This work tackles the inefficiency of attention-based Transformers in long-context, high-difficulty reasoning tasks by introducing PromptCoT-Mamba, an attention-free model built on state-space dual (SSD) layers from Mamba-2. It achieves constant-time, fixed-memory inference per token, avoiding key-value caching, while attaining competitive and, on several benchmarks, state-of-the-art reasoning performance through a two-phase curriculum fine-tuning regime based on the PromptCoT synthesis paradigm. Key contributions include the SSD-based architecture with per-step inference, a scalable training formulation via structured contractions, and a principled data-centric training pipeline that combines foundational reasoning data with rationale-guided PromptCoT examples. The results show PromptCoT-Mamba-7B outperforming comparable Transformer and hybrid baselines and surpassing larger models on tasks like AIME and LiveCodeBench, indicating that state-space models can offer efficient, scalable alternatives for complex reasoning in resource-constrained settings.

Abstract

Large language models (LLMs) have made significant advances in complex reasoning tasks, yet they remain bottlenecked by two core challenges: architectural inefficiency due to reliance on Transformers, and a lack of structured fine-tuning for high-difficulty domains. We introduce \ourmodel, an attention-free language model that addresses both issues through architectural and data-centric innovations. Built on the state space dual (SSD) layers of Mamba-2, our model eliminates the need for self-attention and key-value caching, enabling fixed-memory, constant-time inference. To train it for complex reasoning, we propose a two-phase curriculum fine-tuning strategy based on the \textsc{PromptCoT} synthesis paradigm, which generates pedagogically structured problems via abstract concept selection and rationale-guided generation. On benchmark evaluations, \ourmodel-7B outperforms strong Transformer and hybrid models of comparable scale, and even surpasses the much larger Gemma3-27B by 2.6\% on AIME 24, 0.6\% on AIME 25, and 3.0\% on Livecodebench. These results highlight the potential of state space models as efficient and scalable alternatives to attention-based architectures for high-capacity reasoning.

Paper Structure

This paper contains 20 sections, 5 equations, 2 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Method
  3. State Space Dual Layers in Mamba-2
  4. Inference.
  5. Training.
  6. Curriculum Fine-Tuning with PromptCoT
  7. Experiments
  8. Benchmarks
  9. Evaluation Metrics
  10. Baselines
  11. Implementation Details
  12. Main Results
  13. Discussions
  14. Ablation Study for RQ1
  15. Mathematical Specialization for RQ2
  16. ...and 5 more sections

Figures (2)

  • Figure 1: Comparison of benchmark performance across models of varying architectures. PromptCoT-Mamba-7B, a pure attention-free Mamba model (orange), outperforms all Transformer (blue) and hybrid Mamba-Transformer (hatched) baselines of similar or larger scale on AIME 24, AIME 25, and Livecodebench, while remaining competitive across other math and code reasoning benchmarks. Bars represent pass@1 accuracy.
  • Figure 2: Token generation throughput (tokens/sec) of PromptCoT-Mamba-7B and s1.1-7B under different GPU memory configurations. Performance is measured across a range of generation lengths.