Decompose and Transfer: CoT-Prompting Enhanced Alignment for Open-Vocabulary Temporal Action Detection

Abstract

Open-Vocabulary Temporal Action Detection (OV-TAD) aims to classify and localize action segments in untrimmed videos for unseen categories. Previous methods rely solely on global alignment between label-level semantics and visual features, which is insufficient to transfer temporal consistent visual knowledge from seen to unseen classes. To address this, we propose a Phase-wise Decomposition and Alignment (PDA) framework, which enables fine-grained action pattern learning for effective prior knowledge transfer. Specifically, we first introduce the CoT-Prompting Semantic Decomposition (CSD) module, which leverages the chain-of-thought (CoT) reasoning ability of large language models to automatically decompose action labels into coherent phase-level descriptions, emulating human cognitive processes. Then, Text-infused Foreground Filtering (TIF) module is introduced to adaptively filter action-relevant segments for each phase leveraging phase-wise semantic cues, producing semantically aligned visual representations. Furthermore, we propose the Adaptive Phase-wise Alignment (APA) module to perform phase-level visual-textual matching, and adaptively aggregates alignment results across phases for final prediction. This adaptive phase-wise alignment facilitates the capture of transferable action patterns and significantly enhances generalization to unseen actions. Extensive experiments on two OV-TAD benchmarks demonstrated the superiority of the proposed method.

Figures (9)

• Figure 1: Illustrative examples of seen class LongJump and unseen class PoleVault. (a) The “Start” and “Middle” phases of seen LongJump and unseen PoleVault share strong semantic similarities, despite low label similarity. (b) In light of phase-wise prior knowledge, our method shows higher performance on PoleVault when trained with LongJump.
• Figure 2: An overview of the proposed framework, which comprises three key modules: CoT-Prompting Semantic Decomposition (CSD), Text-infused Foreground Filtering (TIF) and Adaptive Phase-wise Alignment (APA). Specifically, the CSD module temporally decomposes action labels into multiple phase descriptions $F_{t}^{p}$, where $p \in \{start, mid, end, glob\}$. The TIF module then leverages phase-specific semantic cues to adaptively filter action-relevant segments for each phase, yielding phase-specific visual representations $F_{v}^{p}$. Finally, the APA module performs phase-wise alignment and adaptively aggregates the alignment results for final action detection.
• Figure 3: Phase-wise Semantic Similarity ((a)-(d)) and Per-unseen class AP (%) (e) at tIoU threshold 0.5 on THUMOS14 under the 50% seen / 50% unseen split. For (a)-(d), the vertical denotes unseen (testing) classes, the horizontal denotes seen (training) classes.
• Figure 4: Visualization of the detection results on THUMOS14 under the 50% seen / 50% unseen split. “sim” represents the visual-textual similarity at each phase.
• Figure 5: Evaluation Template for GPT-4V and Human Assessments: Rating Across Five Dimensions—Linguistic Quality, Semantic Accuracy, Phase Clarity and Coherence, Visual Alignment, and Transferability.