Zero-shot Action Localization via the Confidence of Large Vision-Language Models

TL;DR

This work tackles zero-shot action localization in long-form videos by introducing ZEAL, a training-free framework that uses a large language model to decompose actions into start and end descriptors and a large vision-language model to evaluate frame-level questions. It combines CLIP-based actionness, per-frame LVLM confidences, and a simple interval-scoring mechanism to generate action proposals without task-specific training. The approach achieves competitive, and often state-of-the-art, zero-shot results on THUMOS14, with clear gains when incorporating video-level labels and LVLM variants, while also validating the importance of boundary confidences over frame-level actionness. ZEAL demonstrates a flexible, scalable path for fine-grained localization in the wild, marking a step toward truly generalizable zero-shot action localization with pretrained multimodal models.

Abstract

Precise action localization in untrimmed video is vital for fields such as professional sports and minimally invasive surgery, where the delineation of particular motions in recordings can dramatically enhance analysis. But in many cases, large scale datasets with video-label pairs for localization are unavailable, limiting the opportunity to fine-tune video-understanding models. Recent developments in large vision-language models (LVLM) address this need with impressive zero-shot capabilities in a variety of video understanding tasks. However, the adaptation of LVLMs, with their powerful visual question answering capabilities, to zero-shot localization in long-form video is still relatively unexplored. To this end, we introduce a true Zero-shot Action Localization method (ZEAL). Specifically, we leverage the built-in action knowledge of a large language model (LLM) to inflate actions into detailed descriptions of the archetypal start and end of the action. These descriptions serve as queries to LVLM for generating frame-level confidence scores which can be aggregated to produce localization outputs. The simplicity and flexibility of our method lends it amenable to more capable LVLMs as they are developed, and we demonstrate remarkable results in zero-shot action localization on a challenging benchmark, without any training. Our code is publicly available at $\href{https://github.com/josaklil-ai/zeal}{github.com/josaklil-ai/zeal}$.

Figures (5)

• Figure 1: Our framework is a true zero-shot method for action localization in long-form, untrimmed video. We leverage off-the-shelf LLM and LVLM for localizing arbitrary actions without needing to optimize any task-specific modules.
• Figure 2: A depiction of the different stages of our framework. Stage 1) Action class names are provided to the LLM and decomposed into three queries: a start query, and end query, and a short description. These queries are yes/no answerable questions that the LVLM uses to assess the likelihood that an activity is starting or ending. Stage 2) To reduce the query search space, we can use a LVLM or CLIP to rank all action classes given key frames from a given video to determine which action classes are most likely to be depicted. Only these selected action classes are considered in the rest of the pipeline. Stage 3) We extract frame-wise CLIP similarity scores with the derived action descriptions to supplement interval filtering when proposals are generated. Stage 4) The queries generated by LLM are then used to prompt the LVLM with each frame of video, and confidence scores are extracted at the frame level. Stage 5) Finally, the distribution of confidence scores are used to generated action interval proposals.
• Figure 3: Precision and recall for action class filtering as a function of number of uniformly sampled frames from a particular video. We plot these for $k=5$ (gray) and $k=3$ (black) and note that even with selecting the 3 most likely classes renders adequate recall for generating LVLM queries.
• Figure 4: Visualization of soft-scores associated with $q^s$ (blue) and $q^e$ (orange) for a video in the THUMOS14 test set, where the ground truth action are "Hammer Throw". In some cases, there is a striking distribution peak where the LVLM is confidently affirmative in answering $q^s$ (such as around second 96) and $q^e$ (seconds 104-108). At 1 fps, the distribution of these scores has a rough alignment to the beginning and end of the ground truth actions. The light gray lines overlaying the barplots indicate the chosen start and end candidates (based on top-$p\%$ threshold). The predictions $\hat{a}$ are constructed from matching start and end candidates as described in Section \ref{['stage5']}. Highly confident predicted intervals are outlined for illustration, and we show the first 2 minutes of the video for clarity.
• Figure 5: Varying values of the $\lambda$ parameter (a tradeoff between localization confidence and CLIP-based actionness, where SigLIP-SoViT-400M/14 is the CLIP) versus downstream mAP performance on THUMOS14. We hold $\sigma$ to a constant value to observe the tradeoff.