Video-OPD: Efficient Post-Training of Multimodal Large Language Models for Temporal Video Grounding via On-Policy Distillation

TL;DR

The paper addresses Temporal Video Grounding (TVG) in multimodal large language models by replacing sparse, episode-level rewards with dense token-level supervision via on-policy distillation. Video-OPD preserves training–inference alignment by sampling trajectories from the current policy and uses a fixed frontier teacher to provide per-token guidance through the reverse KL divergence $D_{KL}(\pi_\theta(\cdot|s_t)\|\pi_{tea}(\cdot|s_t))$, complemented by the TVDF curriculum that prioritizes informative, teacher-reliable trajectories. Empirical results show Video-OPD outperforms GRPO with substantially faster convergence and lower computational cost across standard TVG benchmarks, and TVDF further enhances efficiency. The approach generalizes to broader video understanding tasks, highlighting on-policy distillation as a practical and scalable alternative to traditional RL for long-horizon TVG problems.

Abstract

Reinforcement learning has emerged as a principled post-training paradigm for Temporal Video Grounding (TVG) due to its on-policy optimization, yet existing GRPO-based methods remain fundamentally constrained by sparse reward signals and substantial computational overhead. We propose Video-OPD, an efficient post-training framework for TVG inspired by recent advances in on-policy distillation. Video-OPD optimizes trajectories sampled directly from the current policy, thereby preserving alignment between training and inference distributions, while a frontier teacher supplies dense, token-level supervision via a reverse KL divergence objective. This formulation preserves the on-policy property critical for mitigating distributional shift, while converting sparse, episode-level feedback into fine-grained, step-wise learning signals. Building on Video-OPD, we introduce Teacher-Validated Disagreement Focusing (TVDF), a lightweight training curriculum that iteratively prioritizes trajectories that are both teacher-reliable and maximally informative for the student, thereby improving training efficiency. Empirical results demonstrate that Video-OPD consistently outperforms GRPO while achieving substantially faster convergence and lower computational cost, establishing on-policy distillation as an effective alternative to conventional reinforcement learning for TVG.

Figures (7)

• Figure 1: Limitations of Supervised Fine-Tuning (SFT) and Group Relative Policy Optimization (GRPO) on Temporal Video Grounding (TVG). Blue crowns denote strengths, while red crowns indicate weaknesses. SFT provides dense supervision but is restricted to off-policy optimization, whereas GRPO enables on-policy optimization at the cost of sparse reward signals and multiple rollouts.
• Figure 2: Overview of the Video-OPD post-training framework. Video-OPD optimizes trajectories sampled on-policy to maintain training–inference alignment, leverages a fixed frontier teacher to provide dense token-level supervision via reverse KL for fine-grained credit assignment, and eliminates multiple rollouts per sample, substantially reducing computational overhead.
• Figure 3: Overview of the Teacher-Validated Disagreement Focusing (TVDF) training curriculum. TVDF iteratively prioritizes trajectories that are both teacher-reliable and maximally informative for the student, thereby improving training efficiency.
• Figure 4: Video-OPD on broader video understanding tasks.
• Figure 5: Performance of Video-OPD under multi-round training.