STEP: Enhancing Video-LLMs' Compositional Reasoning by Spatio-Temporal Graph-guided Self-Training

TL;DR

STEP addresses the limitation of Video-LLMs in compositional reasoning by learning from self-generated, reasoning-rich data. It introduces Spatio-Temporal Scene Graphs (STSG) through symbolic structure induction and employs a stepwise, graph-guided rationale learning process to create QA pairs with explicit Chain-of-Thought rationales, trained via a joint loss $L = L_{answer} + \lambda L_{rationale}$ with $\lambda=1$. The approach is model-agnostic and relies on raw videos, achieving up to $21.3\%$ improvements on tasks requiring three or more reasoning steps and demonstrating strong generalization to standard and long-video benchmarks. This work highlights scalable, reasoning-centric data generation for Video-LLMs and broad potential across diverse video understanding tasks.

Abstract

Video Large Language Models (Video-LLMs) have recently shown strong performance in basic video understanding tasks, such as captioning and coarse-grained question answering, but struggle with compositional reasoning that requires multi-step spatio-temporal inference across object relations, interactions, and events. The hurdles to enhancing this capability include extensive manual labor, the lack of spatio-temporal compositionality in existing data and the absence of explicit reasoning supervision. In this paper, we propose STEP, a novel graph-guided self-training method that enables Video-LLMs to generate reasoning-rich fine-tuning data from any raw videos to improve itself. Specifically, we first induce Spatio-Temporal Scene Graph (STSG) representation of diverse videos to capture fine-grained, multi-granular video semantics. Then, the STSGs guide the derivation of multi-step reasoning Question-Answer (QA) data with Chain-of-Thought (CoT) rationales. Both answers and rationales are integrated as training objective, aiming to enhance model's reasoning abilities by supervision over explicit reasoning steps. Experimental results demonstrate the effectiveness of STEP across models of varying scales, with a significant 21.3\% improvement in tasks requiring three or more reasoning steps. Furthermore, it achieves superior performance with a minimal amount of self-generated rationale-enriched training samples in both compositional reasoning and comprehensive understanding benchmarks, highlighting the broad applicability and vast potential.

Figures (6)

• Figure 1: (a) Top left: A significant performance gap between standard understanding and compositional reasoning tasks for advanced Video-LLMs. (b) Top right: Notable improvement with our method. (c) Bottom: An example illustrating the challenging tasks and our performance gains.
• Figure 2: A high-level overview of our STEP approach. We first perform symbolic structure induction to convert spatio-temporal details into a unified STSG. Then a graph-driven rationale learning process is implemented to generate QA pairs with CoT rationales from reasoning paths, providing explicit supervision during training.
• Figure 3: Examples of the process that construct STSG and generate Question-Rationale-Answer (QRA) samples
• Figure 4: The generated STSG accuracy (see measured details in Appendix D) and training loss of STEP and Distillation.
• Figure 5: Example outputs of the model trained by STEP