Boosting Point-supervised Temporal Action Localization via Text Refinement and Alignment

TL;DR

This work tackles point-supervised temporal action localization (PTAL) by addressing the lack of textual semantics in prior vision-only approaches. It introduces Text Refinement and Alignment (TRA), comprising a Point-based Text Refinement (PTR) module to edit captioned descriptions using point annotations and an action-entity memory, and a Point-based Multimodal Alignment (PMA) module to map visual and textual features into a common semantic space with contrastive learning. The training objective combines the base detector loss with a multimodal alignment loss, $\mathcal{L}_{total} = \mathcal{L}_{base} + \lambda \mathcal{L}_{pma}$, where $\mathcal{L}_{pma}$ uses cross-modal losses $\mathcal{L}_{t2v}$ and $\mathcal{L}_{v2t}$. Extensive experiments on THUMOS'14, GTEA, BEOID, ActivityNet1.2, and ActivityNet1.3 demonstrate significant improvements over state-of-the-art PTAL methods, and the pipeline remains practical for single-GPU deployment. The results underline the value of integrating refined textual descriptions and cross-modal alignment to boost localization precision in resource-constrained settings.

Abstract

Recently, point-supervised temporal action localization has gained significant attention for its effective balance between labeling costs and localization accuracy. However, current methods only consider features from visual inputs, neglecting helpful semantic information from the text side. To address this issue, we propose a Text Refinement and Alignment (TRA) framework that effectively utilizes textual features from visual descriptions to complement the visual features as they are semantically rich. This is achieved by designing two new modules for the original point-supervised framework: a Point-based Text Refinement module (PTR) and a Point-based Multimodal Alignment module (PMA). Specifically, we first generate descriptions for video frames using a pre-trained multimodal model. Next, PTR refines the initial descriptions by leveraging point annotations together with multiple pre-trained models. PMA then projects all features into a unified semantic space and leverages a point-level multimodal feature contrastive learning to reduce the gap between visual and linguistic modalities. Last, the enhanced multi-modal features are fed into the action detector for precise localization. Extensive experimental results on five widely used benchmarks demonstrate the favorable performance of our proposed framework compared to several state-of-the-art methods. Moreover, our computational overhead analysis shows that the framework can run on a single 24 GB RTX 3090 GPU, indicating its practicality and scalability.

Figures (6)

• Figure 1: Previous approaches (upper branch) directly use visual features as the input to the detector, whereas our method (lower branch) first generates visual descriptions, then edits them for improved accuracy, and finally aligns multiple modalities as input to the detector.
• Figure 2: Overview of the proposed TRA. Differing from conventional methods that depend solely on visual features (such as RGB and optical flow), our method first employs a pre-trained captioner to generate relevant descriptions, then refines these descriptions through PTR, and maps all features to the same dimension using a linear layer. Subsequently, the detector’s predicted pseudo action and background points are used to select the corresponding features for contrastive learning, achieving feature alignment. Ultimately, the aligned features are fed into the action detector for localization. The detailed structure of PTR is provided in Sec. \ref{['ptr']}.
• Figure 3: The proposed PTR module. (a) shows entity replacement, and (b) represents entity removal.
• Figure 4: Using ChatGPT-4 to construct the action-entity mapping.
• Figure 5: Performance comparison (Average mAP for IoU thresholds of 0.1:0.7 ) under different entity detection thresholds.