Walk Before You Run! Concise LLM Reasoning via Reinforcement Learning
Mingyang Song, Mao Zheng
TL;DR
This work tackles the inefficiency and overthinking of long CoT reasoning in LLMs by introducing ConciseR, a two-stage RL framework. The first stage (GRPO++) strengthens reasoning through clip-higher, dynamic sampling, and an entropy bonus, while the second stage (L-GRPO) enforces conciseness via a length-aware reward. A rule-based reward model is also proposed to prevent reward hacking by basing feedback on verifiable final accuracy and enforcing explicit reasoning-reporting formats. Across benchmarks, ConciseR achieves shorter, more concise CoT with maintained or improved accuracy, delivering practical gains in computation, latency, and resource usage. This approach offers a principled path to efficient, reliable reasoning in long-CoT regimes without sacrificing performance.
Abstract
As test-time scaling becomes a pivotal research frontier in Large Language Models (LLMs) development, contemporary and advanced post-training methodologies increasingly focus on extending the generation length of long Chain-of-Thought (CoT) responses to enhance reasoning capabilities toward DeepSeek R1-like performance. However, recent studies reveal a persistent overthinking phenomenon in state-of-the-art reasoning models, manifesting as excessive redundancy or repetitive thinking patterns in long CoT responses. To address this issue, in this paper, we propose a simple yet effective two-stage reinforcement learning framework for achieving concise reasoning in LLMs, named ConciseR. Specifically, the first stage, using more training steps, aims to incentivize the model's reasoning capabilities via Group Relative Policy Optimization with clip-higher and dynamic sampling components (GRPO++), and the second stage, using fewer training steps, explicitly enforces conciseness and improves efficiency via Length-aware Group Relative Policy Optimization (L-GRPO). Significantly, ConciseR only optimizes response length once all rollouts of a sample are correct, following the "walk before you run" principle. Extensive experimental results demonstrate that our ConciseR model, which generates more concise CoT reasoning responses, outperforms recent state-of-the-art reasoning models with zero RL paradigm across AIME 2024, MATH-500, AMC 2023, Minerva, and Olympiad benchmarks.