One-Stage Open-Vocabulary Temporal Action Detection Leveraging Temporal Multi-scale and Action Label Features

TL;DR

This work tackles Open-vocabulary Temporal Action Detection by proposing a one-stage framework that unifies localization and labeling through Multi-scale Video Analysis (MVA) and Video-Text Alignment (VTA). By processing temporal information at multiple scales and aligning video segments with text concepts, the method reduces error propagation typical of two-stage pipelines and handles actions with diverse durations. Empirical results on THUMOS14 and ActivityNet-1.3 show strong gains in open-vocab settings and competitive performance in closed-vocab settings, validating the effectiveness of combining temporal multi-scale features with cross-modal alignment. The study demonstrates the practical potential of open-vocabulary video understanding and points to future work in incorporating richer context and more accurate start-end timing.

Abstract

Open-vocabulary Temporal Action Detection (Open-vocab TAD) is an advanced video analysis approach that expands Closed-vocabulary Temporal Action Detection (Closed-vocab TAD) capabilities. Closed-vocab TAD is typically confined to localizing and classifying actions based on a predefined set of categories. In contrast, Open-vocab TAD goes further and is not limited to these predefined categories. This is particularly useful in real-world scenarios where the variety of actions in videos can be vast and not always predictable. The prevalent methods in Open-vocab TAD typically employ a 2-stage approach, which involves generating action proposals and then identifying those actions. However, errors made during the first stage can adversely affect the subsequent action identification accuracy. Additionally, existing studies face challenges in handling actions of different durations owing to the use of fixed temporal processing methods. Therefore, we propose a 1-stage approach consisting of two primary modules: Multi-scale Video Analysis (MVA) and Video-Text Alignment (VTA). The MVA module captures actions at varying temporal resolutions, overcoming the challenge of detecting actions with diverse durations. The VTA module leverages the synergy between visual and textual modalities to precisely align video segments with corresponding action labels, a critical step for accurate action identification in Open-vocab scenarios. Evaluations on widely recognized datasets THUMOS14 and ActivityNet-1.3, showed that the proposed method achieved superior results compared to the other methods in both Open-vocab and Closed-vocab settings. This serves as a strong demonstration of the effectiveness of the proposed method in the TAD task.

Figures (4)

• Figure 1: Comparing the 2-stage approach with the proposed method. (a) The 2-stage approach involves localizing temporal actions through proposal generation and utilizing the identified intervals for action identification in the alignment stage. (b) The proposed method leverages multi-scale features for both video-text alignment and action localization.
• Figure 2: (a) Overview of the proposed method. Each video frame is passed through a Pre-trained Image/Video Encoder, followed by Multi-scale Video Analysis and a Decoder for segment detection. Text labels are embedded using a Pre-trained Text Encoder and aligned with the video features to comprehend the relationship and accurately determine the action labels. (b) Each layer in the Multi-scale component utilizes a Transformer Encoder for feature extraction, followed by depthwise 1D convolution for downsampling between layers.
• Figure 3: Change of $\lambda$ coefficient in the overall loss function.
• Figure 4: Illustrating the results of Open-vocab TAD using CLIP B/16 as the text feature.