Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

FROST: Filtering Reasoning Outliers with Attention for Efficient Reasoning

Haozheng Luo, Zhuolin Jiang, Md Zahid Hasan, Yan Chen, Soumalya Sarkar

TL;DR

FROST introduces an attention-aware approach to efficient reasoning by identifying and pruning reasoning outliers—uncritical steps characterized by low attention and low entropy—through a Softmax_1 activation. By replacing the standard Softmax with Softmax_1 and applying lightweight supervised fine-tuning with LoRA, FROST sharpens attention on critical reasoning traces while suppressing uninformative ones, preserving or enhancing reasoning capacity. Empirically, it achieves up to a 26.70% accuracy improvement and a 69.68% reduction in token usage across multiple backbones and benchmarks, along with reductions in attention-outlier metrics such as the maximum infinity norm by 15.97% and average kurtosis by 91.09%. The method generalizes to new reasoning domains (coding and physics) and offers practical gains in both training and deployment efficiency, with a clear theoretical basis for sentence-level outlier suppression via Softmax_1.

Abstract

We propose FROST, an attention-aware method for efficient reasoning. Unlike traditional approaches, FROST leverages attention weights to prune uncritical reasoning paths, yielding shorter and more reliable reasoning trajectories. Methodologically, we introduce the concept of reasoning outliers and design an attention-based mechanism to remove them. Theoretically, FROST preserves and enhances the model's reasoning capacity while eliminating outliers at the sentence level. Empirically, we validate FROST on four benchmarks using two strong reasoning models (Phi-4-Reasoning and GPT-OSS-20B), outperforming state-of-the-art methods such as TALE and ThinkLess. Notably, FROST achieves an average 69.68% reduction in token usage and a 26.70% improvement in accuracy over the base model. Furthermore, in evaluations of attention outlier metrics, FROST reduces the maximum infinity norm by 15.97% and the average kurtosis by 91.09% compared to the base model. Code is available at https://github.com/robinzixuan/FROST

FROST: Filtering Reasoning Outliers with Attention for Efficient Reasoning

TL;DR

FROST introduces an attention-aware approach to efficient reasoning by identifying and pruning reasoning outliers—uncritical steps characterized by low attention and low entropy—through a Softmax_1 activation. By replacing the standard Softmax with Softmax_1 and applying lightweight supervised fine-tuning with LoRA, FROST sharpens attention on critical reasoning traces while suppressing uninformative ones, preserving or enhancing reasoning capacity. Empirically, it achieves up to a 26.70% accuracy improvement and a 69.68% reduction in token usage across multiple backbones and benchmarks, along with reductions in attention-outlier metrics such as the maximum infinity norm by 15.97% and average kurtosis by 91.09%. The method generalizes to new reasoning domains (coding and physics) and offers practical gains in both training and deployment efficiency, with a clear theoretical basis for sentence-level outlier suppression via Softmax_1.

Abstract

We propose FROST, an attention-aware method for efficient reasoning. Unlike traditional approaches, FROST leverages attention weights to prune uncritical reasoning paths, yielding shorter and more reliable reasoning trajectories. Methodologically, we introduce the concept of reasoning outliers and design an attention-based mechanism to remove them. Theoretically, FROST preserves and enhances the model's reasoning capacity while eliminating outliers at the sentence level. Empirically, we validate FROST on four benchmarks using two strong reasoning models (Phi-4-Reasoning and GPT-OSS-20B), outperforming state-of-the-art methods such as TALE and ThinkLess. Notably, FROST achieves an average 69.68% reduction in token usage and a 26.70% improvement in accuracy over the base model. Furthermore, in evaluations of attention outlier metrics, FROST reduces the maximum infinity norm by 15.97% and the average kurtosis by 91.09% compared to the base model. Code is available at https://github.com/robinzixuan/FROST
Paper Structure (41 sections, 3 theorems, 8 equations, 9 figures, 5 tables)

This paper contains 41 sections, 3 theorems, 8 equations, 9 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Reasoning Models.
  4. Efficient Reasoning Methods.
  5. Reasoning Outlier
  6. Attention Distribution of Reasoning Traces
  7. Impact of Reasoning Trace Components on Answer Prediction
  8. Defining and Characterizing Reasoning Outliers
  9. Theoretical Analysis.
  10. FROST
  11. Theoretical Analysis
  12. Setup.
  13. Experimental Studies
  14. Models.
  15. Datasets.
  16. ...and 26 more sections

Key Result

Lemma 5.1

If $\phi$ is monotone coordinatewise, then for any $i,j$, $\bigl(\forall t\in S_i, \exists\, t'\in S_j:\ z_t\ge z_{t'}\bigr)\ \Longrightarrow\ s_i\ \ge\ s_j.$ Consequently, by Assumption ass:softmax1(P1), $\alpha_i=\sigma_1(s)_i \ge \sigma_1(s)_j=\alpha_j$.

Figures (9)

  • Figure 1: The Example of The GPT-OSS-20B Model.
  • Figure 2: Attention Heatmap of Reasoning Tokens. We use the Phi-4-Reasoning model abdin2025phi to generate a reasoning trace for a sample GSM8K question cobbe2021training. The figure shows attention heatmaps from transformer layers 0, 30 and 39, with the first head (top row) and last head (bottom row). Yellow indicates higher attention weights and blue indicates lower ones. In shallow layers, contributions to the final answer are nearly uniform, while deeper layers and later heads highlight specific tokens with stronger influence.
  • Figure 3: Total attention weight distribution to the final answer token </think> from different components of the reasoning trace. We visualize the total attention weight distribution of the Phi-4-Reasoning model on a sample GSM8K question, using transformer layers $1$, $30$, and $40$. The results show that a few reasoning traces contribute strongly to the final token </think>, while many traces have negligible influence, particularly in the layers $30$ and $40$.
  • Figure 4: Theoretical Analysis of Reasoning Outlier Removal. We conduct a theoretical analysis with Phi-4-Reasoning model to demonstrate that removing reasoning outliers using the $\mathop{\rm{Softmax}}_1$ function ( FROST) can preserve or even enhance the model's reasoning capacity. As shown in the figure, the attention weight distribution before and after outlier removal indicates that the model's focus on critical reasoning traces is maintained or improved, while the influence of outliers is significantly reduced.
  • Figure 5: Overview of the FROST workflow We replace the vanilla $\mathop{\rm{Softmax}}$ layer with an outlier-removal layer based on $\mathop{\rm{Softmax}}_1$, followed by SFT to adapt model parameters to the new activation function. We observe that our method significantly reduces the number of low-attention sentences.
  • ...and 4 more figures

Theorems & Definitions (6)

  • Lemma 5.1: Monotone pooling preserves sentence dominance
  • proof
  • Theorem 5.1: $\mathop{\rm{Softmax}}_1$ suppresses sentence-level attention outliers
  • proof
  • Theorem 5.2: Deployment-time suppression of low-attention sentences
  • proof