Klear-Reasoner: Advancing Reasoning Capability via Gradient-Preserving Clipping Policy Optimization

TL;DR

Klear-Reasoner tackles the challenge of long-form reasoning by marrying a quality-focused long Chain-of-Thought supervised fine-tuning stage with Gradient-Preserving clipping Policy Optimization for reinforcement learning. The approach preserves gradient signals from clipped tokens, enabling richer exploration and more robust learning from negative samples, while data curation and soft rewards further stabilize RL and reduce sparsity. Empirical results show strong performance on math and coding benchmarks, matching or surpassing state-of-the-art models of similar scale, especially under extended inference budgets. The work demonstrates that careful data selection, token-level learning signals, and principled RL design can substantially boost long-horizon reasoning capabilities in large models.

Abstract

We present Klear-Reasoner, a model with long reasoning capabilities that demonstrates careful deliberation during problem solving, achieving outstanding performance across multiple benchmarks. Although there are already many excellent works related to inference models in the current community, there are still many problems with reproducing high-performance inference models due to incomplete disclosure of training details. This report provides an in-depth analysis of the reasoning model, covering the entire post-training workflow from data preparation and long Chain-of-Thought supervised fine-tuning (long CoT SFT) to reinforcement learning (RL), along with detailed ablation studies for each experimental component. For SFT data, our experiments show that a small number of high-quality data sources are more effective than a large number of diverse data sources, and that difficult samples can achieve better results without accuracy filtering. In addition, we investigate two key issues with current clipping mechanisms in RL: Clipping suppresses critical exploration signals and ignores suboptimal trajectories. To address these challenges, we propose Gradient-Preserving clipping Policy Optimization (GPPO) that gently backpropagates gradients from clipped tokens. GPPO not only enhances the model's exploration capacity but also improves its efficiency in learning from negative samples. Klear-Reasoner exhibits exceptional reasoning abilities in mathematics and programming, scoring 90.5% on AIME 2024, 83.2% on AIME 2025, 66.0% on LiveCodeBench V5 and 58.1% on LiveCodeBench V6.

Figures (5)

• Figure 1: Benchmark accuracy of Klear-Reasoner-8B on AIME 2024/2025 (avg@64), LiveCodeBench V5 (2024/08/01-2025/02/01, avg@8), and v6 (2025/02/01-2025/05/01, avg@8).
• Figure 2: Comparison of GPPO, GRPO w/ Clip Higher, and CISPO in mathematical RL training. Both methods are trained from an earlier long-CoT SFT checkpoint with a sequence length of 32K tokens. For GRPO, we use the Clip-Higher strategy from DAPO with the recommended $\epsilon_h = 0.28$.
• Figure 3: Comparison between soft reward and hard reward strategies in code RL. Models are initialized from an early long CoT SFT checkpoint and trained for 100 steps with a sequence length of 16K tokens. In the soft reward setting, the reward equals the test case pass rate; in the hard reward setting, a positive reward is given only if all test cases pass and a negative reward otherwise.
• Figure 4: Comparison of code RL performance on LiveCodeBench V5 (avg@4) using filtered versus unfiltered data. The filtering criterion retains prompts with estimated $pass@16 \geq 0.5$, based on 16 completions generated by DeepSeek-R1-0120. Models are initialized from an early-stage SFT checkpoint and trained for 100 steps with a sequence length of 16k tokens. Filter denotes RL results with filtered data, while w/o Filter represents results with the original unfiltered dataset.
• Figure 5: Comparison of math RL performance on AIME2024 (avg@32) with and without filtering of zero-advantage groups. Experiments start from an intermediate RL checkpoint, training with a sequence length of 32K tokens. adv=0 Filter denotes RL training excluding groups where all responses have zero advantage, while w/o adv=0 Filter retains all sampled results regardless of advantage values. Notably, to maintain experimental rigor and ensure a fair comparison, we deliberately avoid supplementing the training data through dynamic sampling methods like those proposed by DBLP:journals/corr/abs-2503-14476.