Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Video Understanding: Through A Temporal Lens

Thong Thanh Nguyen

TL;DR

This thesis investigates how explicit temporal modeling can improve video understanding by focusing on motion relations, event dynamics, and long-form context. It introduces a cohesive set of temporally oriented methods and architectural strategies, including MAMA for robust video-language representation, READ for parameter-efficient temporal transfer in low-resource settings, GSMT with SSL for long-form video QA, motion-aware contrastive learning for temporal panoptic scene graphs, and multi-scale contrastive learning for video grounding. It also presents a temporal-oriented LVLM recipe that analyzes and augments the vision-language interface, memory mechanisms, and scaling strategies to enhance temporal reasoning in large models. Collectively, the work demonstrates that explicit temporal cues significantly boost performance across video-language tasks and offers practical guidance for building scalable, temporally aware video understanding systems. The findings have broad implications for streaming video analysis, long-form content comprehension, and cross-modal reasoning in real-world applications.

Abstract

This thesis explores the central question of how to leverage temporal relations among video elements to advance video understanding. Addressing the limitations of existing methods, the work presents a five-fold contribution: (1) an automatic annotation framework that utilizes large vision-language models and a noise-robust contrastive learning objective with a subtractive angular margin; (2) a parameter-efficient fine-tuning strategy using "recurrent adapters" to capture temporal dynamics in low-data regimes; (3) the integration of State Space Layers (SSL) for efficient long-form video modeling, supported by the introduction of two new long-term benchmarks for egocentric and feature-length content; (4) a novel contrastive learning framework designed to explicitly model fine-grained relations between motions and video moments; and (5) a comprehensive empirical study on Large Vision-Language Models (LVLMs) that identifies the visual-language interface as a bottleneck for temporal reasoning, leading to a new "temporal-oriented recipe" for upscaled video understanding. Collectively, these contributions demonstrate that explicit temporal modeling significantly enhances a model's ability to represent and reason about the fluid nature of video content.

Video Understanding: Through A Temporal Lens

TL;DR

This thesis investigates how explicit temporal modeling can improve video understanding by focusing on motion relations, event dynamics, and long-form context. It introduces a cohesive set of temporally oriented methods and architectural strategies, including MAMA for robust video-language representation, READ for parameter-efficient temporal transfer in low-resource settings, GSMT with SSL for long-form video QA, motion-aware contrastive learning for temporal panoptic scene graphs, and multi-scale contrastive learning for video grounding. It also presents a temporal-oriented LVLM recipe that analyzes and augments the vision-language interface, memory mechanisms, and scaling strategies to enhance temporal reasoning in large models. Collectively, the work demonstrates that explicit temporal cues significantly boost performance across video-language tasks and offers practical guidance for building scalable, temporally aware video understanding systems. The findings have broad implications for streaming video analysis, long-form content comprehension, and cross-modal reasoning in real-world applications.

Abstract

This thesis explores the central question of how to leverage temporal relations among video elements to advance video understanding. Addressing the limitations of existing methods, the work presents a five-fold contribution: (1) an automatic annotation framework that utilizes large vision-language models and a noise-robust contrastive learning objective with a subtractive angular margin; (2) a parameter-efficient fine-tuning strategy using "recurrent adapters" to capture temporal dynamics in low-data regimes; (3) the integration of State Space Layers (SSL) for efficient long-form video modeling, supported by the introduction of two new long-term benchmarks for egocentric and feature-length content; (4) a novel contrastive learning framework designed to explicitly model fine-grained relations between motions and video moments; and (5) a comprehensive empirical study on Large Vision-Language Models (LVLMs) that identifies the visual-language interface as a bottleneck for temporal reasoning, leading to a new "temporal-oriented recipe" for upscaled video understanding. Collectively, these contributions demonstrate that explicit temporal modeling significantly enhances a model's ability to represent and reason about the fluid nature of video content.
Paper Structure (117 sections, 1 theorem, 57 equations, 29 figures, 56 tables, 3 algorithms)

This paper contains 117 sections, 1 theorem, 57 equations, 29 figures, 56 tables, 3 algorithms.

Table of Contents

  1. Introduction
  2. Challenges of Video Understanding
  3. Research Objectives
  4. Contributions and Outline of This Thesis
  5. Literature Review
  6. Video Understanding Tasks
  7. Challenges of Video Understanding
  8. Model Architecture for Video- Language Understanding
  9. Pre-Transformer Architecture
  10. Transformer-based Architecture
  11. LLM-Augmented Architecture
  12. Architecture Analysis
  13. Model Training for Video Understanding
  14. Pre-training for Video Understanding
  15. Fine-tuning for Video Understanding
  16. ...and 102 more sections

Key Result

Theorem 1

Let $\lambda_{i,j}$ denote the angle between the representation of two samples $i, j$, $\mathcal{L}^{v,t}_{\textup{angular}, i}$ and $\mathcal{L}^{v,t}_{\textup{contrastive}, i}$ denote the training objectives with and without the angular margin, respectively. Then, if $\lambda_{i,i} \leq \frac{\pi}

Figures (29)

  • Figure 1: Outline of the thesis.
  • Figure 2: Level hierarchy of video understanding tasks.
  • Figure 3: Timeline of the established video understanding methods (TVR: Text-video retrieval, VC: video captioning, VQA: video question answering, TF: Transformer, LLM: large language model). From left to right, our legend table follows the order: pre-Transformer (Pre-TF), task-specific Transformer, multi-task Transformer, and LLM-augmented architectures.
  • Figure 4: Illustration of text-video retrieval, video captioning, and video question answer (videoQA) tasks.
  • Figure 5: More illustration of video moment retrieval, action recognition, action segmentation, dense video captioning, video chapter generation, and multimodal abstractive summarization tasks.
  • ...and 24 more figures

Theorems & Definitions (1)

  • Theorem 1