Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Text-driven Online Action Detection

Manuel Benavent-Lledo, David Mulero-Pérez, David Ortiz-Perez, Jose Garcia-Rodriguez

TL;DR

This paper addresses online action detection by proposing TOAD, a text-driven architecture that leverages CLIP textual embeddings to initialize a frozen classifier for RGB video features, enabling zero-shot and few-shot learning. TOAD uses a 6-layer transformer video encoder and a CLIP-based textual encoder, with an optional future-action anticipation branch that jointly optimizes current and future predictions via $\mathcal{L} = CE(p, y) + \lambda CE(\hat{p}, \hat{y})$. The method achieves state-of-the-art results on THUMOS14 and sets new baselines for zero-shot and few-shot scenarios on THUMOS14 and TVSeries, while acknowledging weaker performance on TVSeries due to RGB-only input and hardware constraints. Overall, TOAD demonstrates that text-driven, vision-language priors can provide data-efficient online action detection with competitive accuracy and clear avenues for extending to multimodal motion cues and model explainability.

Abstract

Detecting actions as they occur is essential for applications like video surveillance, autonomous driving, and human-robot interaction. Known as online action detection, this task requires classifying actions in streaming videos, handling background noise, and coping with incomplete actions. Transformer architectures are the current state-of-the-art, yet the potential of recent advancements in computer vision, particularly vision-language models (VLMs), remains largely untapped for this problem, partly due to high computational costs. In this paper, we introduce TOAD: a Text-driven Online Action Detection architecture that supports zero-shot and few-shot learning. TOAD leverages CLIP (Contrastive Language-Image Pretraining) textual embeddings, enabling efficient use of VLMs without significant computational overhead. Our model achieves 82.46% mAP on the THUMOS14 dataset, outperforming existing methods, and sets new baselines for zero-shot and few-shot performance on the THUMOS14 and TVSeries datasets.

Text-driven Online Action Detection

TL;DR

This paper addresses online action detection by proposing TOAD, a text-driven architecture that leverages CLIP textual embeddings to initialize a frozen classifier for RGB video features, enabling zero-shot and few-shot learning. TOAD uses a 6-layer transformer video encoder and a CLIP-based textual encoder, with an optional future-action anticipation branch that jointly optimizes current and future predictions via . The method achieves state-of-the-art results on THUMOS14 and sets new baselines for zero-shot and few-shot scenarios on THUMOS14 and TVSeries, while acknowledging weaker performance on TVSeries due to RGB-only input and hardware constraints. Overall, TOAD demonstrates that text-driven, vision-language priors can provide data-efficient online action detection with competitive accuracy and clear avenues for extending to multimodal motion cues and model explainability.

Abstract

Detecting actions as they occur is essential for applications like video surveillance, autonomous driving, and human-robot interaction. Known as online action detection, this task requires classifying actions in streaming videos, handling background noise, and coping with incomplete actions. Transformer architectures are the current state-of-the-art, yet the potential of recent advancements in computer vision, particularly vision-language models (VLMs), remains largely untapped for this problem, partly due to high computational costs. In this paper, we introduce TOAD: a Text-driven Online Action Detection architecture that supports zero-shot and few-shot learning. TOAD leverages CLIP (Contrastive Language-Image Pretraining) textual embeddings, enabling efficient use of VLMs without significant computational overhead. Our model achieves 82.46% mAP on the THUMOS14 dataset, outperforming existing methods, and sets new baselines for zero-shot and few-shot performance on the THUMOS14 and TVSeries datasets.
Paper Structure (15 sections, 4 equations, 2 figures, 6 tables)

This paper contains 15 sections, 4 equations, 2 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. Vision-Language Models
  4. Online Action Detection
  5. TOAD Architecture
  6. Video Encoder
  7. Leveraging Textual Embeddings for Classification
  8. Future Anticipation Facilitates Online Action Detection
  9. Training
  10. Experiments
  11. Experimental Setup
  12. Comparison with State-of-the-Art
  13. Zero-shot & Few-shot Online Action Detection
  14. Ablation Study
  15. Conclusion

Figures (2)

  • Figure 1: Comparison of action recognition (blue), online action detection (orange), and action detection (green). Black segments represent the annotated actions. Note that online action detection may involve multiple actions, background frames without annotated actions, and/or incomplete actions. In contrast, action recognition focuses on classifying predefined annotated segments. And action detection aims to localize and classify actions in the video after it has been fully viewed. Video frames extracted from de2016online.
  • Figure 2: Overview of TOAD. Video features are extracted from downsampled frames (blue) and temporally aggregated using a video transformer encoder (yellow). The current action is predicted by computing the dot product between the video embedding and the textual embedding of the current action, obtained from CLIP's textual encoder (green), ensuring alignment with visual features. Since the representation of future actions differs from that of the ongoing action, a fully connected layer refines the current video representation before computing the dot product with the future action's text prompt (orange). During training, feature extractors for visual and textual features remain frozen.