Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Video-Skill-CoT: Skill-based Chain-of-Thoughts for Domain-Adaptive Video Reasoning

Daeun Lee, Jaehong Yoon, Jaemin Cho, Mohit Bansal

TL;DR

Video-SkoT tackles the challenge of domain-adaptive video reasoning by automatically constructing skill-aware chain-of-thought supervision and routing to skill-specific expert adapters. It automatically builds a skill taxonomy, annotates training data with skill-conditioned multi-step CoT traces, and trains modular LoRA-based experts to specialize in subsets of reasoning skills. Empirical results across ET-Bench, VSI-Bench, and CinePile show consistent gains over strong baselines, with ablations and human evaluation supporting the effectiveness and interpretability of skill-guided reasoning. This approach enhances transferability to unseen domains and provides scalable, interpretable reasoning traces for complex video QA tasks.

Abstract

Recent advances in Chain-of-Thought (CoT) reasoning have improved complex video understanding, but existing methods often struggle to adapt to domain-specific skills (e.g., event detection, spatial relation understanding, emotion understanding) over various video content. To address this, we propose Video-Skill-CoT (a.k.a. Video-SKoT), a framework that automatically constructs and leverages skill-aware CoT supervisions for domain-adaptive video reasoning. First, we construct skill-based CoT annotations: we extract domain-relevant reasoning skills from training questions, cluster them into a shared skill taxonomy, and create detailed multi-step CoT rationale tailored to each video-question pair for training. Second, we introduce a skill-specific expert learning framework. Each expert module specializes in a subset of reasoning skills and is trained with lightweight adapters using the collected CoT supervision. We demonstrate the effectiveness of the proposed approach on three video understanding benchmarks, where Video-SKoT consistently outperforms strong baselines. We also provide in-depth analyses on comparing different CoT annotation pipelines and learned skills over multiple video domains.

Video-Skill-CoT: Skill-based Chain-of-Thoughts for Domain-Adaptive Video Reasoning

TL;DR

Video-SkoT tackles the challenge of domain-adaptive video reasoning by automatically constructing skill-aware chain-of-thought supervision and routing to skill-specific expert adapters. It automatically builds a skill taxonomy, annotates training data with skill-conditioned multi-step CoT traces, and trains modular LoRA-based experts to specialize in subsets of reasoning skills. Empirical results across ET-Bench, VSI-Bench, and CinePile show consistent gains over strong baselines, with ablations and human evaluation supporting the effectiveness and interpretability of skill-guided reasoning. This approach enhances transferability to unseen domains and provides scalable, interpretable reasoning traces for complex video QA tasks.

Abstract

Recent advances in Chain-of-Thought (CoT) reasoning have improved complex video understanding, but existing methods often struggle to adapt to domain-specific skills (e.g., event detection, spatial relation understanding, emotion understanding) over various video content. To address this, we propose Video-Skill-CoT (a.k.a. Video-SKoT), a framework that automatically constructs and leverages skill-aware CoT supervisions for domain-adaptive video reasoning. First, we construct skill-based CoT annotations: we extract domain-relevant reasoning skills from training questions, cluster them into a shared skill taxonomy, and create detailed multi-step CoT rationale tailored to each video-question pair for training. Second, we introduce a skill-specific expert learning framework. Each expert module specializes in a subset of reasoning skills and is trained with lightweight adapters using the collected CoT supervision. We demonstrate the effectiveness of the proposed approach on three video understanding benchmarks, where Video-SKoT consistently outperforms strong baselines. We also provide in-depth analyses on comparing different CoT annotation pipelines and learned skills over multiple video domains.

Paper Structure

This paper contains 37 sections, 1 equation, 12 figures, 9 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Video Understanding with MLLMs.
  4. Skill-specific Expert Learning.
  5. Video-Skill-CoT
  6. Problem Setup
  7. Skill-based CoT Annotation
  8. Step 1: Skill Description & Clustering (\ref{['fig:main1']} Right-(a)).
  9. Step 2: Skill-based CoT Collection (\ref{['fig:main1']} Right-(b)).
  10. Skill-specific Expert Learning
  11. Training Objective.
  12. Experiments
  13. Experiment Setups
  14. Implementation Details.
  15. Datasets and Baselines.
  16. ...and 22 more sections

Figures (12)

  • Figure 1: Left: Video datasets require different reasoning skills. Right: Video-SkoT that automatically constructs and leverages skill-aware CoT supervisions for domain-adaptive video reasoning.
  • Figure 2: Comparison of CoT annotations: (a) regular CoT and (b) our skill-based CoT. Additional examples are provided in Appendix \ref{['sec:add_qual_results']}.
  • Figure 3: Inference output comparison: (a) LLaVA-Video trained with regular CoT and (b) LLaVA-Video trained with our skill-based CoT.Video-SkoT successfully generates temporally grounded and precise rationales that more effectively support accurate answer generation.
  • Figure 4: Skill selection results of VSI-Bench (1)
  • Figure 5: Skill selection results of VSI-Bench (2)
  • ...and 7 more figures