Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

OpenRFT: Adapting Reasoning Foundation Model for Domain-specific Tasks with Reinforcement Fine-Tuning

Yuxiang Zhang, Yuqi Yang, Jiangming Shu, Yuhang Wang, Jinlin Xiao, Jitao Sang

TL;DR

OpenRFT tackles the challenge of domain-specific reasoning with limited annotated data by integrating three data-efficient components: data augmentation, reasoning-process synthesis via a teacher for SFT, and reinforcement learning guided by a Process Reward Model. By casting learning as a language-augmented RL problem and leveraging few-shot in-context knowledge embedding, it demonstrates notable gains on SciKnowEval—averaging about 11% improvement with only 100 domain-specific samples. The work also analyzes critical factors such as the alignment between teacher and student action spaces, the impact of data augmentation, and the benefits of RL supervision over pure SFT, while acknowledging that stronger generalist reasoning foundations and reward-design are needed for broader generalization. Overall, OpenRFT provides a practical, data-efficient pathway to adapt reasoning-founded systems to domain tasks, with public code and datasets to spur further development.

Abstract

OpenAI's recent introduction of Reinforcement Fine-Tuning (RFT) showcases the potential of reasoning foundation model and offers a new paradigm for fine-tuning beyond simple pattern imitation. This technical report presents \emph{OpenRFT}, our attempt to fine-tune generalist reasoning models for domain-specific tasks under the same settings as RFT. OpenRFT addresses two key challenges of lacking reasoning step data and the limited quantity of training samples, by leveraging the domain-specific samples in three ways: question augmentation, synthesizing reasoning-process data, and few-shot ICL. The evaluation is conducted on SciKnowEval, where OpenRFT achieves notable performance gains with only $100$ domain-specific samples for each task. More experimental results will be updated continuously in later versions. Source codes, datasets, and models are disclosed at: https://github.com/ADaM-BJTU/OpenRFT

OpenRFT: Adapting Reasoning Foundation Model for Domain-specific Tasks with Reinforcement Fine-Tuning

TL;DR

OpenRFT tackles the challenge of domain-specific reasoning with limited annotated data by integrating three data-efficient components: data augmentation, reasoning-process synthesis via a teacher for SFT, and reinforcement learning guided by a Process Reward Model. By casting learning as a language-augmented RL problem and leveraging few-shot in-context knowledge embedding, it demonstrates notable gains on SciKnowEval—averaging about 11% improvement with only 100 domain-specific samples. The work also analyzes critical factors such as the alignment between teacher and student action spaces, the impact of data augmentation, and the benefits of RL supervision over pure SFT, while acknowledging that stronger generalist reasoning foundations and reward-design are needed for broader generalization. Overall, OpenRFT provides a practical, data-efficient pathway to adapt reasoning-founded systems to domain tasks, with public code and datasets to spur further development.

Abstract

OpenAI's recent introduction of Reinforcement Fine-Tuning (RFT) showcases the potential of reasoning foundation model and offers a new paradigm for fine-tuning beyond simple pattern imitation. This technical report presents \emph{OpenRFT}, our attempt to fine-tune generalist reasoning models for domain-specific tasks under the same settings as RFT. OpenRFT addresses two key challenges of lacking reasoning step data and the limited quantity of training samples, by leveraging the domain-specific samples in three ways: question augmentation, synthesizing reasoning-process data, and few-shot ICL. The evaluation is conducted on SciKnowEval, where OpenRFT achieves notable performance gains with only domain-specific samples for each task. More experimental results will be updated continuously in later versions. Source codes, datasets, and models are disclosed at: https://github.com/ADaM-BJTU/OpenRFT

Paper Structure

This paper contains 21 sections, 3 equations, 4 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Method
  3. Data Augmentation
  4. SFT-based Imitation
  5. Reasoning Process Synthesis
  6. SFT of Policy Model
  7. RL-based Exploration and Self-improvement
  8. Few-shot ICL-based Domain Knowledge Embedding
  9. RL with PRM
  10. Experiments
  11. Datasets
  12. Comparison Methods
  13. Experimental Setup
  14. Results
  15. Discussions
  16. ...and 6 more sections

Figures (4)

  • Figure 1: OpenRFT framework.
  • Figure 2: Task instructions for generating distinct expressions
  • Figure 3: Performance with different sizes of domain-specific data. The light green dashed line represents the performance of SFT with 100 samples.
  • Figure 4: Reasoning Flaw in Domain-Specific Data: This example shows a chemical equilibrium problem. The final answer is correct, but the red font highlights errors in reasoning. The incorrect calculation of molar mass in Step 2 leads to a wrong intermediate result, although the final answer is still obtained. If process supervision is used, the above problems can be effectively avoided.