Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

ReasonAct: Progressive Training for Fine-Grained Video Reasoning in Small Models

Jiaxin Liu, Zhaolu Kang

TL;DR

The paper addresses the challenge of enabling fine-grained temporal video reasoning in small multimodal models under limited compute. It proposes ReasonAct, a three-stage progressive training framework—Foundational Reasoning Enhancement (FRE), Video-Specific Chain-of-Thought Fine-tuning (V-SFT), and Temporal-Aware Reinforcement Learning (TARL)—augmented by a biomechanically-motivated sub-action decomposition and an enhanced T-GRPO with temporal consistency. Key contributions include constructing a large sub-action library across three datasets, a semantic similarity-based sub-action detector, and a multi-component reward design that jointly optimizes task accuracy, reasoning quality, and temporal coherence. Experiments on HMDB51, UCF-101, and Kinetics-400 show that a 3B-parameter ReasonAct model closes a substantial portion of the gap to larger models, achieving 67.2%, 94.1%, and 78.9% accuracy and improving baselines by up to 18 points, while maintaining reasonable efficiency. The results suggest that explicit reasoning foundations and fine-grained action semantics are practical pathways to deploy capable video reasoning in resource-constrained settings.

Abstract

While recent multimodal models have shown progress in vision-language tasks, small-scale variants still struggle with the fine-grained temporal reasoning required for video understanding. We introduce ReasonAct, a method that enhances video reasoning in smaller models through a three-stage training process: first building a foundation with text-only reasoning, then fine-tuning on video, and finally refining with temporal-aware reinforcement learning. We build upon Temporal Group Relative Policy Optimization (T-GRPO) by incorporating temporal consistency modeling into policy optimization. We also propose a biomechanically-motivated sub-action decomposition mechanism that provides graduated rewards for constituent action phases. Through experiments on HMDB51, UCF-101, and Kinetics-400, our 3B-parameter model achieves 67.2%, 94.1%, and 78.9% accuracy respectively, demonstrating improvements of 17.9, 15.8, and 12.3 points over baselines. Ablation studies validate that our progressive training enables smaller models to achieve competitive video reasoning performance while maintaining computational efficiency.

ReasonAct: Progressive Training for Fine-Grained Video Reasoning in Small Models

TL;DR

The paper addresses the challenge of enabling fine-grained temporal video reasoning in small multimodal models under limited compute. It proposes ReasonAct, a three-stage progressive training framework—Foundational Reasoning Enhancement (FRE), Video-Specific Chain-of-Thought Fine-tuning (V-SFT), and Temporal-Aware Reinforcement Learning (TARL)—augmented by a biomechanically-motivated sub-action decomposition and an enhanced T-GRPO with temporal consistency. Key contributions include constructing a large sub-action library across three datasets, a semantic similarity-based sub-action detector, and a multi-component reward design that jointly optimizes task accuracy, reasoning quality, and temporal coherence. Experiments on HMDB51, UCF-101, and Kinetics-400 show that a 3B-parameter ReasonAct model closes a substantial portion of the gap to larger models, achieving 67.2%, 94.1%, and 78.9% accuracy and improving baselines by up to 18 points, while maintaining reasonable efficiency. The results suggest that explicit reasoning foundations and fine-grained action semantics are practical pathways to deploy capable video reasoning in resource-constrained settings.

Abstract

While recent multimodal models have shown progress in vision-language tasks, small-scale variants still struggle with the fine-grained temporal reasoning required for video understanding. We introduce ReasonAct, a method that enhances video reasoning in smaller models through a three-stage training process: first building a foundation with text-only reasoning, then fine-tuning on video, and finally refining with temporal-aware reinforcement learning. We build upon Temporal Group Relative Policy Optimization (T-GRPO) by incorporating temporal consistency modeling into policy optimization. We also propose a biomechanically-motivated sub-action decomposition mechanism that provides graduated rewards for constituent action phases. Through experiments on HMDB51, UCF-101, and Kinetics-400, our 3B-parameter model achieves 67.2%, 94.1%, and 78.9% accuracy respectively, demonstrating improvements of 17.9, 15.8, and 12.3 points over baselines. Ablation studies validate that our progressive training enables smaller models to achieve competitive video reasoning performance while maintaining computational efficiency.

Paper Structure

This paper contains 28 sections, 8 equations, 3 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Video Understanding and Temporal Reasoning
  4. Multimodal Language Models and Reasoning
  5. Reinforcement Learning for Language Models
  6. Action Decomposition and Sub-Action Recognition
  7. Methodology
  8. Problem Formulation
  9. Stage 1: Foundational Reasoning Enhancement (FRE)
  10. Stage 2: Video-Specific Chain-of-Thought Fine-tuning
  11. Stage 3: Temporal-Aware Reinforcement Learning
  12. Biomechanically-Motivated Sub-Action Recognition
  13. Sub-Action Library Construction.
  14. Semantic Similarity-Based Detection.
  15. Graduated Reward Structure.
  16. ...and 13 more sections

Figures (3)

  • Figure 1: Example of action reasoning. Baseline Qwen2.5-VL misidentifies golf as baseball swing, FRE incorrectly predicts chopping, while ReasonAct correctly identifies golf through structured reasoning.
  • Figure 2: Overview of the ReasonAct framework showing the three-stage training paradigm and key technical components.
  • Figure 3: Enhanced T-GRPO algorithm flowchart showing temporal consistency modeling and multi-reward integration, compared with standard GRPO.