Causality Matters: How Temporal Information Emerges in Video Language Models

TL;DR

This work challenges the prevailing view that positional encodings drive temporal understanding in VideoLMs, showing that temporal reasoning emerges primarily from a causal attention pathway that aggregates inter-frame information across layers and then integrates it into the query. The authors trace this emergent mechanism, demonstrate its causal nature via attention-knockout analyses, and identify a two-stage temporal processing pipeline: inter-frame construction followed by frame-to-query integration. Based on these insights, they propose two efficiency-focused strategies—staged cross-modal attention and a temporal exit mechanism for KV cache compression—and validate them on TempCompass, NExT-QA, and ActivityNet-QA with strong results. The findings offer a principled direction for improving temporal modeling in VideoLMs and reducing computation and memory costs in long-form video reasoning tasks.

Abstract

Video language models (VideoLMs) have made significant progress in multimodal understanding. However, temporal understanding, which involves identifying event order, duration, and relationships across time, still remains a core challenge. Prior works emphasize positional encodings (PEs) as a key mechanism for encoding temporal structure. Surprisingly, we find that removing or modifying PEs in video inputs yields minimal degradation in the performance of temporal understanding. In contrast, reversing the frame sequence while preserving the original PEs causes a substantial drop. To explain this behavior, we conduct substantial analysis experiments to trace how temporal information is integrated within the model. We uncover a causal information pathway: temporal cues are progressively synthesized through inter-frame attention, aggregated in the final frame, and subsequently integrated into the query tokens. This emergent mechanism shows that temporal reasoning emerges from inter-visual token interactions under the constraints of causal attention, which implicitly encodes temporal structure. Based on these insights, we propose two efficiency-oriented strategies: staged cross-modal attention and a temporal exit mechanism for early token truncation. Experiments on two benchmarks validate the effectiveness of both approaches. To the best of our knowledge, this is the first systematic study of video temporal understanding in VideoLMs, offering insights for future model improvement. Our code is available at https://github.com/ANDgate99/Causality-Matters .

Figures (20)

• Figure 1: Temporal information emerges through a causal attention pathway instead of PEs. Reversing position IDs has little effect, but reversing the frame order while keeping position IDs aligned with the original order changes the output.
• Figure 2: Effect of layer-wise PE ablation. Each point shows the change in ground-truth answer probability when the PE is removed from a layer. An obvious drop appears only at the first layer, while later layers show minimal impact.
• Figure 3: Effect of modality-specific position ID shuffling in the first layer. Shuffling query position IDs leads to significant performance drops, while shuffling video position IDs has minimal effect, highlighting the dominant role of PEs in the textual rather than the intended video modality.
• Figure 4: Effect of reversing position IDs versus frame order. Reversing the frame order results in significantly larger performance drops across all task types, suggesting that temporal understanding is primarily driven by the inherent order of frames rather than by positional encodings.
• Figure 5: Effect of blocking attention from the final token to the input sources. Disabling query access causes major performance drops, while blocking video tokens has little effect. This highlights the dominant role of query tokens in final-stage temporal understanding.