REA-RL: Reflection-Aware Online Reinforcement Learning for Efficient Large Reasoning Models

TL;DR

REA-RL tackles the high inference cost of Large Reasoning Models by introducing a reflection-aware online reinforcement learning framework. It combines a lightweight reflection model for sequential revision with a reflection reward that penalizes non-reflective brevity, enabling parallel sampling alongside online revisions. Empirical results show substantial efficiency gains (about 35% cost reduction) without sacrificing accuracy, and analyses confirm the method maintains reflection on hard problems while reducing it on easy ones. The approach offers a practical pathway to scalable, efficient online reasoning for large-scale reasoning systems.

Abstract

Large Reasoning Models (LRMs) demonstrate strong performance in complex tasks but often face the challenge of overthinking, leading to substantially high inference costs. Existing approaches synthesize shorter reasoning responses for LRMs to learn, but are inefficient for online usage due to the time-consuming data generation and filtering processes. Meanwhile, online reinforcement learning mainly adopts a length reward to encourage short reasoning responses, but tends to lose the reflection ability and harm the performance. To address these issues, we propose REA-RL, which introduces a small reflection model for efficient scaling in online training, offering both parallel sampling and sequential revision. Besides, a reflection reward is designed to further prevent LRMs from favoring short yet non-reflective responses. Experiments show that both methods maintain or enhance performance while significantly improving inference efficiency. Their combination achieves a good balance between performance and efficiency, reducing inference costs by 35% without compromising performance. Further analysis demonstrates that our methods are effective by maintaining reflection frequency for hard problems while appropriately reducing it for simpler ones without losing reflection ability. Codes are available at https://github.com/hexuandeng/REA-RL.

Figures (4)

• Figure 1: Overthinking and non-reflective cases from GSM8k. The left column shows the output of DeepSeek-R1-Distill-Qwen-7B, which reflects eight times before finishing generation. The middle column presents the output after online RL training using length rewards, which only spends 103 tokens in "think" part and no reflection, where an error occurs (underlined). The right column shows our output, which uses a similar budget to R1-7B in reasoning but only performs a single reflection.
• Figure 2: Workflow of REA-RL. The upper section illustrates the baseline GRPO process, which parallel sampling$G$ completions $\{s_1, ..., s_G\}$, and optimize using an accuracy reward and a length reward ($\text{R}_\text{Len}$). The bottom section shows our reflection model ($\text{M}_\text{Reflect}$) performing sequential revision. From the $G$ generated completions, the reflection model identifies and truncates overthinking tokens (red segment), keeps only the preceding segments (yellow segment), and then lets the policy model ($\text{M}_\text{Policy}$) complete the answer generation (blue segment), resulting in the revisions $\{s_1^r, ..., s_G^r\}$, with cases shown in Appendix \ref{['apx:case']}. We also introduce a reflection reward ($\text{R}_\text{Reflect}$) and a refined length reward ($\text{R}_\text{RLen}$). Finally, the advantages $\{a_1, ..., a_G, a_1^r, ..., a_G^r\}$ are calculated based on both the initial and revised completions, and used to train the policy model.
• Figure 3: Accuracy and generation length changes during training on GSM8k. The x-axis represents the training steps, with a maximum of 1800 steps. The left plot shows the accuracy of question answering, and the right plot shows the average token consumption per answer. We present results for the GRPO baseline using only a length reward and our three proposed methods.
• Figure 4: Case Study for REA-RL of online data generation. We illustrate how the parallel sampling and sequential revision parts work. Specifically, the yellow, red, and blue parts in this figure correspond to the tokens of the same colors in Figure \ref{['fig:main']}. Since the yellow parts in both completion and revision are identical, they are shown only once in this figure.