Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

APR: Penalizing Structural Redundancy in Large Reasoning Models via Anchor-based Process Rewards

Kaiyan Chang, Chenwei Zhu, Yingfeng Luo, Yifu Huo, Chenglong Wang, Xiaoqian Liu, Qiaozhi He, Tong Xiao, Zhengtao Yu, Jingbo Zhu

TL;DR

The paper identifies a structural redundancy in large reasoning models called the Answer-Stable Tail (AST), which follows the emergence of a final answer and contributes little information. It introduces the Reasoning Anchor to precisely locate where the final answer stabilizes and proposes Anchor-based Process Reward APR to penalize only the AST, not the pre-anchor reasoning. Using DAPO for policy optimization, APR achieves a favorable accuracy-latency Pareto frontier on five mathematical benchmarks at 1.5B and 7B scales, while requiring fewer RL resources than baselines. This structure-aware reward signals more informative feedback during training, enabling more efficient RL optimization and robust performance gains. The work demonstrates that targeting phase-specific redundancy can significantly improve the efficiency of LRMs without sacrificing accuracy, with practical implications for scalable reasoning systems.

Abstract

Test-Time Scaling (TTS) has significantly enhanced the capabilities of Large Reasoning Models (LRMs) but introduces a critical side-effect known as Overthinking. We conduct a preliminary study to rethink this phenomenon from a fine-grained perspective. We observe that LRMs frequently conduct repetitive self-verification without revision even after obtaining the final answer during the reasoning process. We formally define this specific position where the answer first stabilizes as the Reasoning Anchor. By analyzing pre- and post-anchor reasoning behaviors, we uncover the structural redundancy fixed in LRMs: the meaningless repetitive verification after deriving the first complete answer, which we term the Answer-Stable Tail (AST). Motivated by this observation, we propose Anchor-based Process Reward (APR), a structure-aware reward shaping method that localizes the reasoning anchor and penalizes exclusively the post-anchor AST. Leveraging the policy optimization algorithm suitable for length penalties, our APR models achieved the performance-efficiency Pareto frontier at 1.5B and 7B scales averaged across five mathematical reasoning datasets while requiring significantly fewer computational resources for RL training.

APR: Penalizing Structural Redundancy in Large Reasoning Models via Anchor-based Process Rewards

TL;DR

The paper identifies a structural redundancy in large reasoning models called the Answer-Stable Tail (AST), which follows the emergence of a final answer and contributes little information. It introduces the Reasoning Anchor to precisely locate where the final answer stabilizes and proposes Anchor-based Process Reward APR to penalize only the AST, not the pre-anchor reasoning. Using DAPO for policy optimization, APR achieves a favorable accuracy-latency Pareto frontier on five mathematical benchmarks at 1.5B and 7B scales, while requiring fewer RL resources than baselines. This structure-aware reward signals more informative feedback during training, enabling more efficient RL optimization and robust performance gains. The work demonstrates that targeting phase-specific redundancy can significantly improve the efficiency of LRMs without sacrificing accuracy, with practical implications for scalable reasoning systems.

Abstract

Test-Time Scaling (TTS) has significantly enhanced the capabilities of Large Reasoning Models (LRMs) but introduces a critical side-effect known as Overthinking. We conduct a preliminary study to rethink this phenomenon from a fine-grained perspective. We observe that LRMs frequently conduct repetitive self-verification without revision even after obtaining the final answer during the reasoning process. We formally define this specific position where the answer first stabilizes as the Reasoning Anchor. By analyzing pre- and post-anchor reasoning behaviors, we uncover the structural redundancy fixed in LRMs: the meaningless repetitive verification after deriving the first complete answer, which we term the Answer-Stable Tail (AST). Motivated by this observation, we propose Anchor-based Process Reward (APR), a structure-aware reward shaping method that localizes the reasoning anchor and penalizes exclusively the post-anchor AST. Leveraging the policy optimization algorithm suitable for length penalties, our APR models achieved the performance-efficiency Pareto frontier at 1.5B and 7B scales averaged across five mathematical reasoning datasets while requiring significantly fewer computational resources for RL training.
Paper Structure (56 sections, 18 equations, 5 figures, 7 tables)

This paper contains 56 sections, 18 equations, 5 figures, 7 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Overthinking in LLM Reasoning
  4. Efficient Reasoning via Reinforcement Learning
  5. Preliminary Study
  6. Anchor Definition
  7. Anchor Localization
  8. Rule-based Localization.
  9. Model-based Localization.
  10. Empirical Analysis of Structural Redundancy
  11. Observation 1: Reliability of Anchor Localization Methods.
  12. Observation 2: The "First Answer is Final" Phenomenon.
  13. Observation 3: Redundancy Exists Independently of Correctness.
  14. Observation 4: Verbose Reasoning Structure Impairs Accuracy via Truncation.
  15. Conclusion.
  16. ...and 41 more sections

Figures (5)

  • Figure 1: Schematic diagram of structural redundancy in LRMs. The reasoning anchor splits the trace into an information-dense pre-anchor phase of effective derivation, correction and first conclusion, and an information-sparse post-anchor phase of repetitive self-verification without revision, termed the Answer-Stable Tail (AST).
  • Figure 2: Redundancy Ratios ($\rho$) across four LRMs on various mathematical reasoning datasets.
  • Figure 3: Statistical analysis of reasoning observations.
  • Figure 4: Performance-efficiency Pareto frontier averaged across five datasets.
  • Figure 5: Comparison of Rule-based and Model-based Anchor Localization Methods (Left y-axis: average generation length; Right y-axis: average Pass@1 accuracy, averaged over 16 samples).