RIFT: Repurposing Negative Samples via Reward-Informed Fine-Tuning

TL;DR

RIFT addresses data inefficiency in LLM alignment by reusing all self-generated samples and reweighting them with scalar rewards to learn from both correct and failing outputs. It formalizes a generalized signed-weighted objective, demonstrates instability in naive reward integration, and introduces a stabilized loss with a linear surrogate for negative samples. Empirical results on seven mathematical benchmarks across multiple Qwen base models show that RIFT consistently outperforms SFT, DFT, RFT, and DPO, while reducing memory usage and maintaining training efficiency. The approach enables robust, data-efficient post-training alignment without requiring a reference model, and it extends to reasoner models with substantial gains in Mean@8 and Pass@8. Overall, RIFT offers a practical, scalable pathway to improve alignment by learning from mixed-quality, self-generated data.

Abstract

While Supervised Fine-Tuning (SFT) and Rejection Sampling Fine-Tuning (RFT) are standard for LLM alignment, they either rely on costly expert data or discard valuable negative samples, leading to data inefficiency. To address this, we propose Reward Informed Fine-Tuning (RIFT), a simple yet effective framework that utilizes all self-generated samples. Unlike the hard thresholding of RFT, RIFT repurposes negative trajectories, reweighting the loss with scalar rewards to learn from both the positive and negative trajectories from the model outputs. To overcome the training collapse caused by naive reward integration, where direct multiplication yields an unbounded loss, we introduce a stabilized loss formulation that ensures numerical robustness and optimization efficiency. Extensive experiments on mathematical benchmarks across various base models show that RIFT consistently outperforms RFT. Our results demonstrate that RIFT is a robust and data-efficient alternative for alignment using mixed-quality, self-generated data.

Figures (2)

• Figure 1: Efficiency and performance of post-training methods for Qwen2.5-Math-1.5B fine-tuned on MATH. Left: average accuracy against peak memory utilization (training efficiency); Right: per-dataset accuracy (generalization). RIFT surpasses strong baselines in accuracy while requiring less computational memory.
• Figure 2: A comparative overview of RFT and RIFT. Unlike RFT rejects negative samples and only trains on positive ones, RIFT repurposes negative samples through a unified reward-informed loss. To ensure stable optimization, a linear surrogate is applied to negative samples to prevent loss collapse.

Theorems & Definitions (7)

• Definition 3.1: Naive Signed-Weighted Loss
• Theorem 3.2: Gradient Explosion and Unboundedness
• Definition 3.3: RIFT Loss
• Theorem 3.4: Stability and Properties of RIFT
• proof : Proof of Theorem \ref{['thm:collapse']}
• Theorem A.1: Formal Statement of Theorem \ref{['thm:RIFT_properties']}
• proof : Proof of Theorem \ref{['thm:RIFT_properties']}