Enhancing Video-Language Representations with Structural Spatio-Temporal Alignment

TL;DR

Finnsta is designed as a plug&play system, which can be integrated into existing well-trained VLMs for further representation augmentation, without training from scratch or relying on SG annotations in downstream applications.

Abstract

While pre-training large-scale video-language models (VLMs) has shown remarkable potential for various downstream video-language tasks, existing VLMs can still suffer from certain commonly seen limitations, e.g., coarse-grained cross-modal aligning , under-modeling of temporal dynamics, detached video-language view. In this work, we target enhancing VLMs with a fine-grained structural spatio-temporal alignment learning method (namely Finsta). First of all, we represent the input texts and videos with fine-grained scene graph (SG) structures, both of which are further unified into a holistic SG (HSG) for bridging two modalities. Then, an SG-based framework is built, where the textual SG (TSG) is encoded with a graph Transformer, while the video dynamic SG (DSG) and the HSG are modeled with a novel recurrent graph Transformer for spatial and temporal feature propagation. A spatial-temporal Gaussian differential graph Transformer is further devised to strengthen the sense of the changes in objects across spatial and temporal dimensions. Next, based on the fine-grained structural features of TSG and DSG, we perform object-centered spatial alignment and predicate-centered temporal alignment respectively, enhancing the video-language grounding in both the spatiality and temporality. We design our method as a plug&play system, which can be integrated into existing well-trained VLMs for further representation augmentation, without training from scratch or relying on SG annotations in downstream applications. On 6 representative VL modeling tasks over 12 datasets in both standard and long-form video scenarios, Finsta consistently improves the existing 13 strong-performing VLMs persistently, and refreshes the current state-of-the-art end task performance significantly in both the fine-tuning and zero-shot settings.

Figures (21)

• Figure 1: Representing the text and video with the corresponding scene graphs (SGs) enables more fine-grained control of video-language correspondence learning (same colors denote same concepts), as the SG representations depict the intrinsic modal-agnostic semantic structures of texts or videos. Best viewed in color.
• Figure 2: We represent the input text and video with textual scene graph (TSG) and dynamic scene graph (DSG), respectively. We unify the TSG and DSG into a holistic SG (HSG) by adding the cross-modal coreference edges.
• Figure 3: (a) The high-level view of our fine-grained structural spatio-temporal alignment learning (Finsta) framework based on the dual-stream-sum architecture. And (b) the detailed dataflow of the recurrent graph Transformer (R-GTrm).
• Figure 4: Illustration of the STGD-GTrm for modeling the spatio-temporal changes.
• Figure 5: The predicate-centered temporal contrasting mechanism, where we extract the spatial region and temporal interval for the predicate-centered temporal alignment.