Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Positive-Unlabeled Reinforcement Learning Distillation for On-Premise Small Models

Zhiqiang Kou, Junyang Chen, Xin-Qiang Cai, Xiaobo Xia, Ming-Kun Xie, Dong-Dong Wu, Biao Liu, Yuheng Jia, Xin Geng, Masashi Sugiyama, Tat-Seng Chua

TL;DR

This paper tackles RL-based alignment for on-premise small models by eliminating the need for human preferences or reward models. It introduces Reinforcement Learning Capability Distillation (RLCD) with a PU anchor mechanism: for each prompt, the teacher is queried once to obtain an anchor $a$, while the student locally samples candidates $U(x)$ and builds a PU-aware soft distribution $D_x$ via anchor-conditioned self-evaluation. The LDL-GRPO objective then aligns the group-level policy to $D_x$ while regularizing toward the SFT baseline, with theoretical guarantees of order-consistency and near-optimal concentration. Empirically, LDL-GRPO and AnchorRank-DPO achieve consistent gains across unimodal and multimodal on-premise tasks under constrained teacher calls, enabling a practical SFT-to-RL pipeline for private, latency-sensitive settings. The results demonstrate a scalable, low-cost route to RL-aligned small models that maintain privacy and compliance while delivering improved instruction-following and reasoning capabilities.

Abstract

Due to constraints on privacy, cost, and latency, on-premise deployment of small models is increasingly common. However, most practical pipelines stop at supervised fine-tuning (SFT) and fail to reach the reinforcement learning (RL) alignment stage. The main reason is that RL alignment typically requires either expensive human preference annotation or heavy reliance on high-quality reward models with large-scale API usage and ongoing engineering maintenance, both of which are ill-suited to on-premise settings. To bridge this gap, we propose a positive-unlabeled (PU) RL distillation method for on-premise small-model deployment. Without human-labeled preferences or a reward model, our method distills the teacher's preference-optimization capability from black-box generations into a locally trainable student. For each prompt, we query the teacher once to obtain an anchor response, locally sample multiple student candidates, and perform anchor-conditioned self-ranking to induce pairwise or listwise preferences, enabling a fully local training loop via direct preference optimization or group relative policy optimization. Theoretical analysis justifies that the induced preference signal by our method is order-consistent and concentrates on near-optimal candidates, supporting its stability for preference optimization. Experiments demonstrate that our method achieves consistently strong performance under a low-cost setting.

Positive-Unlabeled Reinforcement Learning Distillation for On-Premise Small Models

TL;DR

This paper tackles RL-based alignment for on-premise small models by eliminating the need for human preferences or reward models. It introduces Reinforcement Learning Capability Distillation (RLCD) with a PU anchor mechanism: for each prompt, the teacher is queried once to obtain an anchor , while the student locally samples candidates and builds a PU-aware soft distribution via anchor-conditioned self-evaluation. The LDL-GRPO objective then aligns the group-level policy to while regularizing toward the SFT baseline, with theoretical guarantees of order-consistency and near-optimal concentration. Empirically, LDL-GRPO and AnchorRank-DPO achieve consistent gains across unimodal and multimodal on-premise tasks under constrained teacher calls, enabling a practical SFT-to-RL pipeline for private, latency-sensitive settings. The results demonstrate a scalable, low-cost route to RL-aligned small models that maintain privacy and compliance while delivering improved instruction-following and reasoning capabilities.

Abstract

Due to constraints on privacy, cost, and latency, on-premise deployment of small models is increasingly common. However, most practical pipelines stop at supervised fine-tuning (SFT) and fail to reach the reinforcement learning (RL) alignment stage. The main reason is that RL alignment typically requires either expensive human preference annotation or heavy reliance on high-quality reward models with large-scale API usage and ongoing engineering maintenance, both of which are ill-suited to on-premise settings. To bridge this gap, we propose a positive-unlabeled (PU) RL distillation method for on-premise small-model deployment. Without human-labeled preferences or a reward model, our method distills the teacher's preference-optimization capability from black-box generations into a locally trainable student. For each prompt, we query the teacher once to obtain an anchor response, locally sample multiple student candidates, and perform anchor-conditioned self-ranking to induce pairwise or listwise preferences, enabling a fully local training loop via direct preference optimization or group relative policy optimization. Theoretical analysis justifies that the induced preference signal by our method is order-consistent and concentrates on near-optimal candidates, supporting its stability for preference optimization. Experiments demonstrate that our method achieves consistently strong performance under a low-cost setting.
Paper Structure (38 sections, 2 theorems, 24 equations, 8 figures, 7 tables, 1 algorithm)

This paper contains 38 sections, 2 theorems, 24 equations, 8 figures, 7 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Preference-Based Alignment
  4. KD for On-Premise Language Models
  5. Self-Rewarded Reinforcement Learning
  6. Method
  7. Preliminaries
  8. Stage I: Teacher-Guided SFT
  9. Stage II: Reinforcement Learning Capability Distillation (RLCD)
  10. Theoretical Justification
  11. Experiments
  12. Setups
  13. Results and Findings
  14. Ablation Study
  15. Convergence Analysis
  16. ...and 23 more sections

Key Result

Theorem 3.1

For any $i,j\in\{1,\dots,K\}$, $r_i>r_j$ implies $D_\mathrm{x}(i)>D_\mathrm{x}(j)$.

Figures (8)

  • Figure 1: Comparison among SFT, RLHF, and anchor-guided self-alignment (ours). For on-premise small-model deployment, training pipelines typically stop at the first-stage SFT: the small model imitates black-box teacher responses but fails to acquire preference-optimization–driven self-improvement. The second-stage RL alignment is hard to realize locally because it relies on large-scale human annotations or extensive reward-model calls. We propose a low-cost on-premise alignment strategy: for each query, we call the black-box teacher once to obtain an anchor and use anchor-guided local sampling and self-evaluation to generate scalar signals, enabling a practical SFT-to-RL loop under strict cost and deployment constraints.
  • Figure 2: Two-stage convergence behavior from SFT to LDL-GRPO. The dashed vertical line marks the transition from SFT (Stage I) to LDL-GRPO (Stage II). Results are shown for CountProb with LLaMA3-8B and A-OKVQA-RG with LLaVA-1.5-7B. Across unimodal and multimodal tasks, LDL-GRPO exhibits stable loss trajectories after the transition, without sudden divergence.
  • Figure 3: Sensitivity analysis on CountProb (LLaMA3-8B). Representative sweeps with fixed $\beta$ (left) and fixed $\tau$ (right).
  • Figure C.4: Prompt used for automatic pairwise preference evaluation.
  • Figure C.5: Two-stage convergence from SFT to LDL-GRPO across models and tasks. Each panel shows one model--task pair, illustrating a stable transition from SFT to LDL-GRPO.
  • ...and 3 more figures

Theorems & Definitions (6)

  • Theorem 3.1: Order consistency
  • proof : Proof sketch
  • Theorem 3.2: Near-optimality on the sampled set
  • proof : Proof sketch
  • proof
  • proof