Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

RLSR: Reinforcement Learning with Supervised Reward Outperforms SFT in Instruction Following

Zhichao Wang, Andy Wong, Ruslan Belkin

TL;DR

RLSR tackles the challenge of aligning large language models to instruction-following without abandoning the extensive SFT data. By defining a reward as the cosine similarity between embedding representations of model outputs and human-labeled responses, RLSR integrates SFT data into a reinforcement-learning framework, enabling exploration beyond token-level supervision. Empirical results show that RLSR commonly surpasses SFT, with the SFT+RLSR pipeline achieving the highest AlpacaEval win rates (e.g., up to 30.73% on Qwen-7B with INFINITY), and performance improvements scale with embedding-reward strength. These findings demonstrate a scalable approach to improve instruction following while leveraging existing high-quality SFT datasets, at the cost of additional compute for RL rollout and reward computation.

Abstract

After the pretraining stage of LLMs, techniques such as SFT, RLHF, RLVR, and RFT are applied to enhance instruction-following ability, mitigate undesired responses, improve reasoning capability and enable efficient domain adaptation with minimal data. SFT relies on the next-token prediction objective to strengthen instruction following in a base model using a large corpus of human-labeled responses. In contrast, RFT employs a RL-based approach to adapt fine-tuned reasoning models to specific domains with limited supervision. Inspired by RFT, we propose replacing SFT with RLSR to leverage the extensive SFT dataset in an RL framework, thereby improving the base model's instruction-following ability. In RLSR, the base model generates multiple responses for each prompt, and reward scores are computed as the cosine similarity in the semantic embedding space between the generated and human-labeled responses. RLSR can be utilized in multiple ways. It can directly replace SFT, achieving superior performance on instruction-following benchmarks-for example, RLSR (SB) on Qwen-7B (INFINITY) achieved an AlpacaEval win rate of 26.34%, surpassing SFT's 21.01%. Furthermore, combining SFT and RLSR further enhances downstream task performance; Qwen-7B (INFINITY) achieved a win rate of 30.73% when trained with SFT + RLSR.

RLSR: Reinforcement Learning with Supervised Reward Outperforms SFT in Instruction Following

TL;DR

RLSR tackles the challenge of aligning large language models to instruction-following without abandoning the extensive SFT data. By defining a reward as the cosine similarity between embedding representations of model outputs and human-labeled responses, RLSR integrates SFT data into a reinforcement-learning framework, enabling exploration beyond token-level supervision. Empirical results show that RLSR commonly surpasses SFT, with the SFT+RLSR pipeline achieving the highest AlpacaEval win rates (e.g., up to 30.73% on Qwen-7B with INFINITY), and performance improvements scale with embedding-reward strength. These findings demonstrate a scalable approach to improve instruction following while leveraging existing high-quality SFT datasets, at the cost of additional compute for RL rollout and reward computation.

Abstract

After the pretraining stage of LLMs, techniques such as SFT, RLHF, RLVR, and RFT are applied to enhance instruction-following ability, mitigate undesired responses, improve reasoning capability and enable efficient domain adaptation with minimal data. SFT relies on the next-token prediction objective to strengthen instruction following in a base model using a large corpus of human-labeled responses. In contrast, RFT employs a RL-based approach to adapt fine-tuned reasoning models to specific domains with limited supervision. Inspired by RFT, we propose replacing SFT with RLSR to leverage the extensive SFT dataset in an RL framework, thereby improving the base model's instruction-following ability. In RLSR, the base model generates multiple responses for each prompt, and reward scores are computed as the cosine similarity in the semantic embedding space between the generated and human-labeled responses. RLSR can be utilized in multiple ways. It can directly replace SFT, achieving superior performance on instruction-following benchmarks-for example, RLSR (SB) on Qwen-7B (INFINITY) achieved an AlpacaEval win rate of 26.34%, surpassing SFT's 21.01%. Furthermore, combining SFT and RLSR further enhances downstream task performance; Qwen-7B (INFINITY) achieved a win rate of 30.73% when trained with SFT + RLSR.
Paper Structure (17 sections, 3 equations, 1 figure, 4 tables)

This paper contains 17 sections, 3 equations, 1 figure, 4 tables.

Table of Contents

  1. Introduction
  2. Methodology
  3. SFT
  4. RLHF, RLVR and RFT
  5. RLSR
  6. Differences between RLSR and RFT
  7. Experiments
  8. Models, Datasets, and Evaluation
  9. Performance Evaluation and Analysis
  10. Comparison Between SFT and RLSR
  11. Comparison Between SFT and SFT+RLSR
  12. Comparison Between RLSR and SFT+RLSR
  13. Comparison Among Different Reward Models
  14. Literature Review
  15. Limitation
  16. ...and 2 more sections

Figures (1)

  • Figure 1: (a) SFT fine-tunes the base model on human-labeled responses to maximize the likelihood of next-token prediction. (b) RFT adapts the reasoning model to specific domains with minimal data with a RL framework, employing a grader function that may optionally incorporate human-labeled responses. (c) RLSR builds upon the RFT framework while leveraging the data used in SFT. It strengthens the base model's instruction-following ability by defining a reward function based on the cosine similarity between the embedding representations of the model-generated and human-labeled responses, utilizing large-scale SFT data.