TeMTG: Text-Enhanced Multi-Hop Temporal Graph Modeling for Audio-Visual Video Parsing

TL;DR

TeMTG tackles the weakly supervised Audio-Visual Video Parsing problem by introducing text-enhanced semantic guidance and a multi-hop temporal graph to explicitly model temporal dependencies across segments. The framework fuses text embeddings from pre-trained multimodal models with audio-visual features using modality-specific MLPs, then reasons over a K-hop temporal graph via multi-head GAT, followed by MMIL pooling to produce per-modality and joint predictions. Key contributions include a text-enhanced multimodal fusion mechanism using pseudo labels from CLAP/CLIP and a K-hop temporal graph architecture that captures both short-term and long-range event continuity. Experiments on the LLP dataset show TeMTG achieving state-of-the-art performance on several segment- and event-level metrics, demonstrating improved semantic alignment and temporal reasoning, with some remaining challenges in AV-boundary precision under weak supervision.

Abstract

Audio-Visual Video Parsing (AVVP) task aims to parse the event categories and occurrence times from audio and visual modalities in a given video. Existing methods usually focus on implicitly modeling audio and visual features through weak labels, without mining semantic relationships for different modalities and explicit modeling of event temporal dependencies. This makes it difficult for the model to accurately parse event information for each segment under weak supervision, especially when high similarity between segmental modal features leads to ambiguous event boundaries. Hence, we propose a multimodal optimization framework, TeMTG, that combines text enhancement and multi-hop temporal graph modeling. Specifically, we leverage pre-trained multimodal models to generate modality-specific text embeddings, and fuse them with audio-visual features to enhance the semantic representation of these features. In addition, we introduce a multi-hop temporal graph neural network, which explicitly models the local temporal relationships between segments, capturing the temporal continuity of both short-term and long-range events. Experimental results demonstrate that our proposed method achieves state-of-the-art (SOTA) performance in multiple key indicators in the LLP dataset.

Figures (2)

• Figure 1: (a) Illustration of the AVVP task. (b) Structure for Multi-Hop Temporal Graph (assume $K=2$).
• Figure 2: TeMTG architecture: Audio and visual features are fused with text embeddings, aggregated, and then processed by multi-hop temporal graphs with multi-head residual GAT layers to model event dependencies.