Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

PCL-Reasoner-V1.5: Advancing Math Reasoning with Offline Reinforcement Learning

Yao Lu, Dengdong Fan, Jianzheng Nie, Fan Xu, Jie Chen, Bin Zhou, Yonghong Tian

TL;DR

This paper tackles the challenge of enhancing mathematical reasoning in large language models while addressing the instability and high cost of online reinforcement learning (RL). It introduces PCL-Reasoner-V1.5, a 32B parameter LLM built on Qwen2.5-32B, fine-tuned with supervised CoT data and refined via an offline RL pipeline that uses a fixed, carefully filtered dataset for policy optimization. Evaluations on AIME 2024 and 2025 show state-of-the-art performance among models post-trained on Qwen2.5-32B, with accuracies of $90.9\%$ and $85.6\%$, respectively, and an emphasis on RL-induced longer, more deliberate long-CoT reasoning. The work demonstrates that offline RL can deliver stable, efficient improvements in reasoning, with publicly released data and code, and suggests this paradigm as a competitive alternative to online RL for advancing reasoning in large language models.

Abstract

We present PCL-Reasoner-V1.5, a 32-billion-parameter large language model (LLM) for mathematical reasoning. The model is built upon Qwen2.5-32B and refined via supervised fine-tuning (SFT) followed by reinforcement learning (RL). A central innovation is our proposed offline RL method, which provides superior training stability and efficiency over standard online RL methods such as GRPO. Our model achieves state-of-the-art performance among models post-trained on Qwen2.5-32B, attaining average accuracies of 90.9% on AIME 2024 and 85.6% on AIME 2025. Our work demonstrates offline RL as a stable and efficient paradigm for advancing reasoning in LLMs. All experiments were conducted on Huawei Ascend 910C NPUs.

PCL-Reasoner-V1.5: Advancing Math Reasoning with Offline Reinforcement Learning

TL;DR

This paper tackles the challenge of enhancing mathematical reasoning in large language models while addressing the instability and high cost of online reinforcement learning (RL). It introduces PCL-Reasoner-V1.5, a 32B parameter LLM built on Qwen2.5-32B, fine-tuned with supervised CoT data and refined via an offline RL pipeline that uses a fixed, carefully filtered dataset for policy optimization. Evaluations on AIME 2024 and 2025 show state-of-the-art performance among models post-trained on Qwen2.5-32B, with accuracies of and , respectively, and an emphasis on RL-induced longer, more deliberate long-CoT reasoning. The work demonstrates that offline RL can deliver stable, efficient improvements in reasoning, with publicly released data and code, and suggests this paradigm as a competitive alternative to online RL for advancing reasoning in large language models.

Abstract

We present PCL-Reasoner-V1.5, a 32-billion-parameter large language model (LLM) for mathematical reasoning. The model is built upon Qwen2.5-32B and refined via supervised fine-tuning (SFT) followed by reinforcement learning (RL). A central innovation is our proposed offline RL method, which provides superior training stability and efficiency over standard online RL methods such as GRPO. Our model achieves state-of-the-art performance among models post-trained on Qwen2.5-32B, attaining average accuracies of 90.9% on AIME 2024 and 85.6% on AIME 2025. Our work demonstrates offline RL as a stable and efficient paradigm for advancing reasoning in LLMs. All experiments were conducted on Huawei Ascend 910C NPUs.
Paper Structure (14 sections, 3 equations, 5 figures, 2 tables)

This paper contains 14 sections, 3 equations, 5 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Online vs. Offline RL
  3. Our Approach
  4. Prompt Template
  5. SFT
  6. Data.
  7. Training.
  8. Offline RL.
  9. Data.
  10. Inference.
  11. Verification & Filtering.
  12. Training.
  13. Evaluation
  14. Conclusion

Figures (5)

  • Figure 1: Evaluation results of 32B models on AIME (pass@1)
  • Figure 2: Comparison between online and offline RL. $\mathcal{X}_i$ denotes a mini-batch of questions and $\mathcal{Y}_i$ denotes the corresponding model-inferred answers. (a) Online RL: An iterative cycle where $\mathcal{Y}_i$ is inferred on-the-fly by the current policy $\pi_{\theta_i}$ and used for immediate training. (b) Offline RL: A sequential process where all answers for the whole question set are inferred once, forming a static dataset for subsequent training.
  • Figure 3: Training pipeline
  • Figure 4: Training loss $L_{\text{norm}}(\theta)$
  • Figure 5: Accuracy on each response length range on AIME 2024 & 2025