Training-Free Zero-Shot Temporal Action Detection with Vision-Language Models

TL;DR

The paper tackles zero-shot temporal action detection without any training by leveraging vision-language models to classify and localize unseen activities directly in untrimmed videos. It introduces FreeZAD, which derives a video-level pseudo-label from ViL embeddings and uses a prototype-guided localization framework with LogOIC and Actionness Calibration to produce confident segment predictions. To further boost performance, AdaZAD adds a test-time adaptation strategy with Prototype-Centric Sampling, refining cross-modal alignment on a per-video basis. Experiments on THUMOS14 and ActivityNet-1.3 show competitive results against unsupervised methods while significantly reducing runtime, and the AdaZAD extension narrows the gap to fully supervised approaches, highlighting the practical value of training-free ViL-based ZSTAD.

Abstract

Existing zero-shot temporal action detection (ZSTAD) methods predominantly use fully supervised or unsupervised strategies to recognize unseen activities. However, these training-based methods are prone to domain shifts and require high computational costs, which hinder their practical applicability in real-world scenarios. In this paper, unlike previous works, we propose a training-Free Zero-shot temporal Action Detection (FreeZAD) method, leveraging existing vision-language (ViL) models to directly classify and localize unseen activities within untrimmed videos without any additional fine-tuning or adaptation. We mitigate the need for explicit temporal modeling and reliance on pseudo-label quality by designing the LOGarithmic decay weighted Outer-Inner-Contrastive Score (LogOIC) and frequency-based Actionness Calibration. Furthermore, we introduce a test-time adaptation (TTA) strategy using Prototype-Centric Sampling (PCS) to expand FreeZAD, enabling ViL models to adapt more effectively for ZSTAD. Extensive experiments on the THUMOS14 and ActivityNet-1.3 datasets demonstrate that our training-free method outperforms state-of-the-art unsupervised methods while requiring only 1/13 of the runtime. When equipped with TTA, the enhanced method further narrows the gap with fully supervised methods.

Figures (6)

• Figure 1: We propose the first training-free method for ZSTAD, distinguishing it from all existing state-of-the-art methods, which are training-based with varying degrees of supervision.
• Figure 2: Overall architecture of our proposed training-free zero-shot temporal action detection network. The model recognizes and localizes unseen activities within untrimmed videos with only a single forward pass. Specifically, video-level label is first generated through visual embedding and textual encoding derived from the visual and textual backbones of ViL models. Then, segments are selected by calculating the similarities between this label and the visual features of each frame through a filtering operation. Finally, the confidence score for each segment is generated using LogOIC and refined by Actionness Calibration.
• Figure 3: An illustration of Logarithmic Decay Weighted Outer-Inner-Contrastive Score (LogOIC). The green mask and the red mask respectively cover the inner and outer activations within a segment. The adjusted outer activation, shown in orange, are obtained by reweighting the blue activation with a logarithmic decay weight.
• Figure 4: Illustration of zero-shot temporal action detection with test-time adaptation (AdaZAD). This paradigm seeks to adapt the pre-trained ViL models for TAD without annotations, and the detected results are obtained after completing all adaptation steps. Prototype-Centric Sampling (PCS) focuses on selecting positive samples surrounding the highest similarity values to the textual features.
• Figure 5: The left plot shows the average mAP across various adaptation steps $M$ , while the right plot presents the average mAP for different tIoU thresholds under three relaxed unsupervised constraints: perfect class, perfect samples, and a combination of both.