OVD: On-policy Verbal Distillation

TL;DR

OVD tackles the memory bottleneck of token-level distillation by replacing full vocabulary logit matching with trajectory-level verbal feedback from teachers. By enabling on-policy learning with verbal rejection sampling and a GRPO-based objective, OVD facilitates interactive supervision and scalable long-horizon reasoning transfer from powerful teachers to smaller students. Theoretical results prove unbiased gradient estimates and convergence under a mixture training distribution, while experiments on Web Q&A and math reasoning show substantial performance gains and improved sample efficiency. The approach yields memory-efficient, black-box-friendly distillation that preserves exploration and demonstrates strong practical impact on challenging reasoning tasks.

Abstract

Knowledge distillation offers a promising path to transfer reasoning capabilities from large teacher models to efficient student models; however, existing token-level on-policy distillation methods require token-level alignment between the student and teacher models, which restricts the student model's exploration ability, prevent effective use of interactive environment feedback, and suffer from severe memory bottlenecks in reinforcement learning. We introduce On-policy Verbal Distillation (OVD), a memory-efficient framework that replaces token-level probability matching with trajectory matching using discrete verbal scores (0--9) from teacher models. OVD dramatically reduces memory consumption while enabling on-policy distillation from teacher models with verbal feedback, and avoids token-level alignment, allowing the student model to freely explore the output space. Extensive experiments on Web question answering and mathematical reasoning tasks show that OVD substantially outperforms existing methods, delivering up to +12.9% absolute improvement in average EM on Web Q&A tasks and a up to +25.7% gain on math benchmarks (when trained with only one random samples), while also exhibiting superior training efficiency. Our project page is available at https://OVD.github.io

Figures (6)

• Figure 1: Memory consumption scales linearly with sequence length $L$ for token-level distillation on Qwen2.5-7B. Logits storage grows $\sim$16$\times$ faster than KV cache per token, making it the primary memory bottleneck for long-context distillation.
• Figure 2: Memory consumption scales linearly with the number of rollouts per problem $N$ at fixed sequence length $L=8192$ for Qwen2.5-7B. Both logits and KV cache grow proportionally to $N$, with logits requiring $\sim$16$\times$ more memory than KV cache. With $N=32$ rollouts, logits alone require 240 GB, making token-level distillation impractical for RL.
• Figure 3: The policy model performs trajectory sampling (left), while the teacher model applies rejection sampling and returns search results for Web Q&A (right).
• Figure 4: Ablation study comparing different simulator training strategies and threshold configurations across seven Q&A benchmarks. Datasets are ordered by increasing difficulty from left to right: NQ, TriviaQA, PopQA (single-hop), HotpotQA, 2Wiki, Musique, Bamboogle (multi-hop). (a) Prompt-based simulators show modest performance with average scores around 0.20-0.24. (b) SFT-based simulators substantially outperform prompt-based approaches, with the best configuration (T10, QR test T10) achieving an average score of 0.486, demonstrating the importance of proper simulator design and threshold alignment.
• Figure 5: Reward convergence on Qwen-2.5-7B across training steps under different rejection thresholds

Theorems & Definitions (8)

• Theorem 3.1: Unbiased Gradient Estimation
• Proposition 3.2: Variance Reduction
• Proposition 3.3: Convergence under Mixture Training
• Proposition 3.4: Score Granularity and Approximation Quality
• proof
• proof
• proof
• proof