Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

NEAT: Neuron-Based Early Exit for Large Reasoning Models

Kang Liu, Yongkang Liu, Xiaocui Yang, Peidong Wang, Wen Zhang, Shi Feng, Yifei Zhang, Daling Wang

TL;DR

NEAT introduces a training-free, neuron-based framework for early exit in large reasoning models by identifying exit-associated neurons through termination-token attribution and late activation, then monitoring their activation dynamics at inference to either exit or suppress reflection. The approach yields 22–28% token reduction across multiple benchmarks while maintaining accuracy and achieving notable latency improvements, outperforming output-based early-exit methods that are calibration-sensitive. By leveraging internal neuron signals instead of external probes or repeated rollouts, NEAT offers a robust, computation-efficient mechanism to mitigate overthinking in large reasoning tasks with broad practical implications for scalable reasoning systems.

Abstract

Large Reasoning Models (LRMs) often suffer from \emph{overthinking}, a phenomenon in which redundant reasoning steps are generated after a correct solution has already been reached. Existing early reasoning exit methods primarily rely on output-level heuristics or trained probing models to skip redundant reasoning steps, thereby mitigating overthinking. However, these approaches typically require additional rollout computation or externally labeled datasets. In this paper, we propose \textbf{NEAT}, a \textbf{N}euron-based \textbf{E}arly re\textbf{A}soning exi\textbf{T} framework that monitors neuron-level activation dynamics to enable training-free early exits, without introducing additional test-time computation. NEAT identifies exit-associated neurons and tracks their activation patterns during reasoning to dynamically trigger early exit or suppress reflection, thereby reducing unnecessary reasoning while preserving solution quality. Experiments on four reasoning benchmarks across six models with different scales and architectures show that, for each model, NEAT achieves an average token reduction of 22\% to 28\% when averaged over the four benchmarks, while maintaining accuracy.

NEAT: Neuron-Based Early Exit for Large Reasoning Models

TL;DR

NEAT introduces a training-free, neuron-based framework for early exit in large reasoning models by identifying exit-associated neurons through termination-token attribution and late activation, then monitoring their activation dynamics at inference to either exit or suppress reflection. The approach yields 22–28% token reduction across multiple benchmarks while maintaining accuracy and achieving notable latency improvements, outperforming output-based early-exit methods that are calibration-sensitive. By leveraging internal neuron signals instead of external probes or repeated rollouts, NEAT offers a robust, computation-efficient mechanism to mitigate overthinking in large reasoning tasks with broad practical implications for scalable reasoning systems.

Abstract

Large Reasoning Models (LRMs) often suffer from \emph{overthinking}, a phenomenon in which redundant reasoning steps are generated after a correct solution has already been reached. Existing early reasoning exit methods primarily rely on output-level heuristics or trained probing models to skip redundant reasoning steps, thereby mitigating overthinking. However, these approaches typically require additional rollout computation or externally labeled datasets. In this paper, we propose \textbf{NEAT}, a \textbf{N}euron-based \textbf{E}arly re\textbf{A}soning exi\textbf{T} framework that monitors neuron-level activation dynamics to enable training-free early exits, without introducing additional test-time computation. NEAT identifies exit-associated neurons and tracks their activation patterns during reasoning to dynamically trigger early exit or suppress reflection, thereby reducing unnecessary reasoning while preserving solution quality. Experiments on four reasoning benchmarks across six models with different scales and architectures show that, for each model, NEAT achieves an average token reduction of 22\% to 28\% when averaged over the four benchmarks, while maintaining accuracy.
Paper Structure (35 sections, 9 equations, 7 figures, 6 tables)

This paper contains 35 sections, 9 equations, 7 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Early Reasoning Exit.
  4. Neurons in LLMs.
  5. Methodology
  6. Exit Neurons Identification
  7. Neuron Attribution.
  8. Temporal Filtering.
  9. Inference-Time Intervention
  10. Activation Pattern Matching
  11. Dynamic Intervention Strategy
  12. Experiments
  13. Experimental Setup
  14. Benchmarks and Metrics.
  15. Baselines.
  16. ...and 20 more sections

Figures (7)

  • Figure 1: Comparison of different early reasoning exit methods. (a) Output-based Answer Heuristic methods (b) Probe-based method. (c) Our method NEAT.
  • Figure 2: Overview of the proposed NEAT framework. Left: During calibration, we identify exit-associated neurons by computing attribution scores at the termination timestep and filtering based on temporal activation patterns. Right: During inference, we monitor the activation dynamics of the identified neurons and apply dynamic intervention when the pattern matches the reference regime.
  • Figure 3: Inference latency (seconds per question) on AIME2024 across Qwen3-series models of different scales.
  • Figure 4: Sensitivity analysis with respect to the number of monitored neurons in $\mathcal{S}^*$. Blue solid lines denote accuracy (left y-axis), and red dashed lines denote average response length (#Tok, right y-axis), where the right axis is inverted. Horizontal dotted lines indicate the Vanilla baseline. (a) Qwen3-8B. (b) DeepSeek-R1-Distill-Qwen-7B.
  • Figure 5: Case study illustrating overthinking in Vanilla decoding and how NEAT avoids redundant continuation.
  • ...and 2 more figures