Video-Language Alignment via Spatio-Temporal Graph Transformer

TL;DR

This work tackles video-language alignment by modeling detailed spatio-temporal relationships among vision tokens and exploiting cross-pair similarities across video-text data. It introduces the Spatio-Temporal Graph Transformer (STGT), which embeds 2D spatial and temporal cues into vision tokens, builds a learnable spatio-temporal graph from local token similarities, and integrates graph topology into transformer attention. A cross-similarity alignment loss (CSAL) complements the initial contrastive objective to exploit inherent self-similarity within video-video and text-text pairs, further boosting alignment accuracy. Through a two-stage pre-training regime on large-scale web video-text data and rigorous evaluation on retrieval and video QA benchmarks, STGT achieves state-of-the-art performance, highlighting the value of graph-augmented spatio-temporal context in multi-modal learning and its potential for practical video understanding applications.

Abstract

Video-language alignment is a crucial multi-modal task that benefits various downstream applications, e.g., video-text retrieval and video question answering. Existing methods either utilize multi-modal information in video-text pairs or apply global and local alignment techniques to promote alignment precision. However, these methods often fail to fully explore the spatio-temporal relationships among vision tokens within video and across different video-text pairs. In this paper, we propose a novel Spatio-Temporal Graph Transformer module to uniformly learn spatial and temporal contexts for video-language alignment pre-training (dubbed STGT). Specifically, our STGT combines spatio-temporal graph structure information with attention in transformer block, effectively utilizing the spatio-temporal contexts. In this way, we can model the relationships between vision tokens, promoting video-text alignment precision for benefiting downstream tasks. In addition, we propose a self-similarity alignment loss to explore the inherent self-similarity in the video and text. With the initial optimization achieved by contrastive learning, it can further promote the alignment accuracy between video and text. Experimental results on challenging downstream tasks, including video-text retrieval and video question answering, verify the superior performance of our method.

Figures (15)

• Figure 1: (a) The similarity between vision tokens in video. (b) The spatio-temporal graph. (c) The pair and similarity relationship between video and texts.
• Figure 2: Overview of the proposed framework. Firstly, vision tokens are obtained by a pre-trained ViT. Subsequently, a spatio-temporal graph transformer module is used to enhance global and local features. Finally, a self-similarity alignment loss is implemented to optimize video-text alignment following contrastive learning optimization. The BLIP-2 blip2 attention masking strategy for each objective to control query-text interaction.
• Figure 3: Illustrations of the local spatio-temporal graph transformer.
• Figure 4: Illustration of adjacency matrix generation. (a) Similarity matrix $\mathcal{W}_s$; (b) Spatio-temporal mask $\mathcal{T}_{mask}$; (c) Adjacency matrix $\mathcal{A}$. Each frame is depicted with two blocks. In (b), yellow denotes intra-frame similarity, white denotes inter-frame similarity, and gray denotes masked similarity.
• Figure 5: Illustrations of relationships in videos and texts.