Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Enhancing Temporal Modeling of Video LLMs via Time Gating

Zi-Yuan Hu, Yiwu Zhong, Shijia Huang, Michael R. Lyu, Liwei Wang

TL;DR

The paper tackles the gap in temporal modeling within Video LLMs, proposing Time Gating (TG) to enhance temporal-aware video understanding. TG-Vid inserts a module-specific Time Gating mechanism between a frozen vision encoder and QFormer, gating spatial, temporal, and MLP sub-paths over $T$ video frames and per-frame query length $L_q$. Experiments on MVBench, TempCompass, and NExT-QA show TG-Vid achieving state-of-the-art results, with ablations confirming the effectiveness of the TG design and its components. The work contributes a practical method to improve temporal reasoning in multimodal video-language models, with potential extensions to longer videos and broader video-language tasks.

Abstract

Video Large Language Models (Video LLMs) have achieved impressive performance on video-and-language tasks, such as video question answering. However, most existing Video LLMs neglect temporal information in video data, leading to struggles with temporal-aware video understanding. To address this gap, we propose a Time Gating Video LLM (TG-Vid) designed to enhance temporal modeling through a novel Time Gating module (TG). The TG module employs a time gating mechanism on its sub-modules, comprising gating spatial attention, gating temporal attention, and gating MLP. This architecture enables our model to achieve a robust understanding of temporal information within videos. Extensive evaluation of temporal-sensitive video benchmarks (i.e., MVBench, TempCompass, and NExT-QA) demonstrates that our TG-Vid model significantly outperforms the existing Video LLMs. Further, comprehensive ablation studies validate that the performance gains are attributed to the designs of our TG module. Our code is available at https://github.com/LaVi-Lab/TG-Vid.

Enhancing Temporal Modeling of Video LLMs via Time Gating

TL;DR

The paper tackles the gap in temporal modeling within Video LLMs, proposing Time Gating (TG) to enhance temporal-aware video understanding. TG-Vid inserts a module-specific Time Gating mechanism between a frozen vision encoder and QFormer, gating spatial, temporal, and MLP sub-paths over video frames and per-frame query length . Experiments on MVBench, TempCompass, and NExT-QA show TG-Vid achieving state-of-the-art results, with ablations confirming the effectiveness of the TG design and its components. The work contributes a practical method to improve temporal reasoning in multimodal video-language models, with potential extensions to longer videos and broader video-language tasks.

Abstract

Video Large Language Models (Video LLMs) have achieved impressive performance on video-and-language tasks, such as video question answering. However, most existing Video LLMs neglect temporal information in video data, leading to struggles with temporal-aware video understanding. To address this gap, we propose a Time Gating Video LLM (TG-Vid) designed to enhance temporal modeling through a novel Time Gating module (TG). The TG module employs a time gating mechanism on its sub-modules, comprising gating spatial attention, gating temporal attention, and gating MLP. This architecture enables our model to achieve a robust understanding of temporal information within videos. Extensive evaluation of temporal-sensitive video benchmarks (i.e., MVBench, TempCompass, and NExT-QA) demonstrates that our TG-Vid model significantly outperforms the existing Video LLMs. Further, comprehensive ablation studies validate that the performance gains are attributed to the designs of our TG module. Our code is available at https://github.com/LaVi-Lab/TG-Vid.
Paper Structure (22 sections, 5 equations, 3 figures, 10 tables)

This paper contains 22 sections, 5 equations, 3 figures, 10 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Video Large Language Models.
  4. Video Temporal Modeling.
  5. Methodology
  6. Preliminary
  7. Time Gating Module (TG)
  8. Time Gating Video LLM
  9. Experiments
  10. Main Results
  11. Ablation Studies
  12. TG Module.
  13. Time Gating Mechanism.
  14. TG Components.
  15. Conclusion
  16. ...and 7 more sections

Figures (3)

  • Figure 1: Model architecture of TG-Vid. Given a video with $T$ frames, the vision encoder extracts $T$ frame-level embeddings. Our TG employs a novel time gating mechanism to enhance video temporal modeling, thereby enhancing the frame-level video modeling ability of the QFormer. Moving forward, the QFormer compresses each frame-level video embedding from $L$ patch tokens to $L_\text{q}$ query tokens, followed by LLMs.
  • Figure 2: Ablation study on the number of layers of the TG module based on TG-Vid-197K.
  • Figure 3: Visualization of Time Gating.