Reinforcement Learning Tuning for VideoLLMs: Reward Design and Data Efficiency

TL;DR

This paper introduces Temporal-RLT, a GRPO-based post-training framework to boost video-specific reasoning in multimodal large language models. It employs a dual reward design—discrete semantic signals from multi-choice VideoQA and continuous temporal signals from temporal IoU grounding—together with variance-aware data selection to improve learning efficiency. The authors construct Temporal-RLT-Full-490k and Temporal-RLT-32k datasets and demonstrate strong, data-efficient gains across eight video understanding benchmarks, including VideoQA, grounding, and reasoning tasks, outperforming supervised fine-tuning and prior RLT baselines. The work highlights reward design and sample selection as key levers for scalable, reasoning-centric video understanding with VideoLLMs, while noting limitations and future directions for richer thinking supervision.

Abstract

Understanding real-world videos with complex semantics and long temporal dependencies remains a fundamental challenge in computer vision. Recent progress in multimodal large language models (MLLMs) has demonstrated strong capabilities in vision-language tasks, while reinforcement learning tuning (RLT) has further improved their reasoning abilities. In this work, we explore RLT as a post-training strategy to enhance the video-specific reasoning capabilities of MLLMs. Built upon the Group Relative Policy Optimization (GRPO) framework, we propose a dual-reward formulation that supervises both semantic and temporal reasoning through discrete and continuous reward signals. To facilitate effective preference-based optimization, we introduce a variance-aware data selection strategy based on repeated inference to identify samples that provide informative learning signals. We evaluate our approach across eight representative video understanding tasks, including VideoQA, Temporal Video Grounding, and Grounded VideoQA. Our method consistently outperforms supervised fine-tuning and existing RLT baselines, achieving superior performance with significantly less training data. These results underscore the importance of reward design and data selection in advancing reasoning-centric video understanding with MLLMs. Notably, The initial code release (two months ago) has now been expanded with updates, including optimized reward mechanisms and additional datasets. The latest version is available at https://github.com/appletea233/Temporal-R1 .

Figures (7)

• Figure 1: Overview of our reward-based framework for post-training on video understanding tasks.Top left: Semantic reasoning is supervised using a discrete reward from multi-choice VideoQA, where the model selects the correct answer. Top right: Temporal reasoning is guided by a continuous reward based on temporal IoU in grounding tasks. Bottom left: A reward-based data selection strategy filters training samples based on behavioral variance. High-variance samples induce stronger preference gradients and are prioritized for GRPO optimization. Bottom right: Performance comparison across baselines and our Temporal-RLT, showing consistent improvements on QA, grounding, and reasoning tasks.
• Figure 2: Overview of our training framework. Given a video and its corresponding instruction, the VideoLLM generates either an answer choice for multi-choice QA or a temporal segment for grounding. The model is optimized using a GRPO-based RLT pipeline with task-specific rewards, including a discrete reward for QA and a continuous temporal IoU reward for grounding.
• Figure 3: Data statistic of Temporal-RLT-Full-490k training dataset. We find that converting all answers to an open-ended format is critical in reliably assessing question difficulty and effective model training.
• Figure 4: Changes in data distribution before and after RLT. We report the distribution of correctness counts for MC-VideoQA and the IoU variance (max–min) for temporal grounding. The post-RLT data reflects only the subset of samples selected from the original dataset based on our sampling strategy.
• Figure 5: Visualization of Temporal-RLT's reasoning outputs on the Temporal Video Grounding task. The three examples, from top to bottom, come from Charades-STA, ActivityNet, and ActivityNet-RTL, respectively.