Factorized Learning for Temporally Grounded Video-Language Models

TL;DR

This work tackles the challenge of temporally grounded video-language understanding by introducing D$^2$VLM, a factorized learning framework that decouples temporal evidence grounding from textual answering while maintaining their dependency through an evidence-referencing generation flow. Key innovations include novel <evi> tokens that capture event-level visual semantics for grounding and an explicit consistency mechanism between grounding and answer stages, plus Factorized Preference Optimization (FPO) that integrates probabilistic temporal grounding into preference learning. A synthetic, factorized data synthesis pipeline enables training data with explicit grounding and textual perturbations, addressing data gaps for multi-factor preferences. Empirical results across ET Bench, Charades-STA, and YouCook2 show that D$^2$VLM with FPO achieves state-of-the-art or competitive performance with a relatively compact model, demonstrating the practical viability and generality of factorized learning for temporally grounded video-language tasks.

Abstract

Recent video-language models have shown great potential for video understanding, but still struggle with accurate temporal grounding for event-level perception. We observe that two main factors in video understanding (i.e., temporal grounding and textual response) form a logical hierarchy: accurate temporal evidence grounding lays the foundation for reliable textual response. However, existing works typically handle these two tasks in a coupled manner without a clear logical structure, leading to sub-optimal objectives. We address this from a factorized learning perspective. We first propose D$^2$VLM, a framework that decouples the learning of these two tasks while also emphasizing their inherent dependency. We adopt a "grounding then answering with evidence referencing" paradigm and introduce evidence tokens for evidence grounding, which emphasize event-level visual semantic capture beyond the focus on timestamp representation in existing works. To further facilitate the learning of these two tasks, we introduce a novel factorized preference optimization (FPO) algorithm. Unlike standard preference optimization, FPO explicitly incorporates probabilistic temporal grounding modeling into the optimization objective, enabling preference learning for both temporal grounding and textual response. We also construct a synthetic dataset to address the lack of suitable datasets for factorized preference learning with explicit temporal grounding. Experiments on various tasks demonstrate the clear advantage of our approach. Our source code is available at https://github.com/nusnlp/d2vlm.

Figures (8)

• Figure 1: Qualitative examples for grounding-focused task, dense captioning-related task, and temporally grounded video question answering task.
• Figure 2: Conceptual demonstration of the D$^2$VLM framework.
• Figure 2: An illustrative example of the data synthesis approach.
• Figure 3: The visual semantic capture process of <evi> token.
• Figure 3: A qualitative example for dense captioning task.