Causality-Aware Temporal Projection for Video Understanding in Video-LLMs

TL;DR

The paper tackles the challenge of temporal ordering and causal coherence in video understanding with parameter-efficient Video-LLMs. It introduces V-CORE, comprising Learnable Spatial Aggregation (LSA) and Causality-Aware Temporal Projector (CATP), which enforce unidirectional temporal flow via block-causal attention and a terminal dynamic summary, all trained with 4-bit QLoRA on frozen backbones. Key contributions include explicit temporal constraints at the projection stage, leading to improved temporal and causal reasoning (e.g., 61.2% accuracy on NExT-QA) while maintaining efficiency across benchmarks. The work demonstrates that architectural constraints on temporal reasoning, rather than sheer model capacity, can significantly enhance long-range video understanding in practical, hardware-efficient settings.

Abstract

Recent Video Large Language Models (Video-LLMs) have shown strong multimodal reasoning capabilities, yet remain challenged by video understanding tasks that require consistent temporal ordering and causal coherence. Many parameter-efficient Video-LLMs rely on unconstrained bidirectional projectors to model inter-frame interactions, which can blur temporal ordering by allowing later frames to influence earlier representations, without explicit architectural mechanisms to respect the directional nature of video reasoning. To address this limitation, we propose V-CORE, a parameter-efficient framework that introduces explicit temporal ordering constraints for video understanding. V-CORE consists of two key components: (1) Learnable Spatial Aggregation (LSA), which adaptively selects salient spatial tokens to reduce redundancy, and (2) a Causality-Aware Temporal Projector (CATP), which enforces structured unidirectional information flow via block-causal attention and a terminal dynamic summary token acting as a causal sink. This design preserves intra-frame spatial interactions while ensuring that temporal information is aggregated in a strictly ordered manner. With 4-bit QLoRA and a frozen LLM backbone, V-CORE can be trained efficiently on a single consumer GPU. Experiments show that V-CORE achieves strong performance on the challenging NExT-QA benchmark, reaching 61.2% accuracy, and remains competitive across MSVD-QA, MSRVTT-QA, and TGIF-QA, with gains concentrated in temporal and causal reasoning subcategories (+3.5% and +5.2% respectively), directly validating the importance of explicit temporal ordering constraints.

Figures (4)

• Figure 1: Baseline vs. V-CORE. Baseline: Unconstrained bidirectional attention may mix information across temporal directions. V-CORE: Explicit unidirectional aggregation via a terminal causal sink enforces ordered temporal reasoning.
• Figure 2: Overall architecture of V-CORE. Video frames are spatially compressed into salient tokens $H_t$ via LSA and temporally modeled by CATP under a block-causal constraint. The Causal Sink aggregates the sequence for reasoning in the LLM with LoRA.
• Figure 3: Hyperparameter sensitivity on NExT-QA with respect to the spatial token count $K$ and the frame count $T$.
• Figure 4: Qualitative comparison between the baseline and V-CORE. Each case illustrates how our model leverages temporal constraints and global context to outperform baseline.