Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Zero-Shot Action Recognition in Surveillance Videos

Joao Pereira, Vasco Lopes, David Semedo, Joao Neves

TL;DR

The paper tackles zero-shot action recognition in surveillance videos under data and labeling constraints. It leverages Large Vision-Language Models, specifically VideoLLaMA2, and a novel token-level Self-Reflective Sampling (Self-ReS) strategy to better capture non-linear, long-form events. On the UCF-Crime dataset, the approach yields a significant zero-shot accuracy boost over CLIP baselines (around 20 percentage points), with Self-ReS further elevating performance to 44.6%. This work demonstrates the potential of LVLMs for versatile, low-supervision surveillance video analysis and points to future improvements through in-context learning and enhanced sampling.

Abstract

The growing demand for surveillance in public spaces presents significant challenges due to the shortage of human resources. Current AI-based video surveillance systems heavily rely on core computer vision models that require extensive finetuning, which is particularly difficult in surveillance settings due to limited datasets and difficult setting (viewpoint, low quality, etc.). In this work, we propose leveraging Large Vision-Language Models (LVLMs), known for their strong zero and few-shot generalization, to tackle video understanding tasks in surveillance. Specifically, we explore VideoLLaMA2, a state-of-the-art LVLM, and an improved token-level sampling method, Self-Reflective Sampling (Self-ReS). Our experiments on the UCF-Crime dataset show that VideoLLaMA2 represents a significant leap in zero-shot performance, with 20% boost over the baseline. Self-ReS additionally increases zero-shot action recognition performance to 44.6%. These results highlight the potential of LVLMs, paired with improved sampling techniques, for advancing surveillance video analysis in diverse scenarios.

Zero-Shot Action Recognition in Surveillance Videos

TL;DR

The paper tackles zero-shot action recognition in surveillance videos under data and labeling constraints. It leverages Large Vision-Language Models, specifically VideoLLaMA2, and a novel token-level Self-Reflective Sampling (Self-ReS) strategy to better capture non-linear, long-form events. On the UCF-Crime dataset, the approach yields a significant zero-shot accuracy boost over CLIP baselines (around 20 percentage points), with Self-ReS further elevating performance to 44.6%. This work demonstrates the potential of LVLMs for versatile, low-supervision surveillance video analysis and points to future improvements through in-context learning and enhanced sampling.

Abstract

The growing demand for surveillance in public spaces presents significant challenges due to the shortage of human resources. Current AI-based video surveillance systems heavily rely on core computer vision models that require extensive finetuning, which is particularly difficult in surveillance settings due to limited datasets and difficult setting (viewpoint, low quality, etc.). In this work, we propose leveraging Large Vision-Language Models (LVLMs), known for their strong zero and few-shot generalization, to tackle video understanding tasks in surveillance. Specifically, we explore VideoLLaMA2, a state-of-the-art LVLM, and an improved token-level sampling method, Self-Reflective Sampling (Self-ReS). Our experiments on the UCF-Crime dataset show that VideoLLaMA2 represents a significant leap in zero-shot performance, with 20% boost over the baseline. Self-ReS additionally increases zero-shot action recognition performance to 44.6%. These results highlight the potential of LVLMs, paired with improved sampling techniques, for advancing surveillance video analysis in diverse scenarios.

Paper Structure

This paper contains 8 sections, 1 equation, 3 figures, 1 table.

Table of Contents

  1. Introduction
  2. Methodology
  3. Large Vision and Language Models.
  4. Sampling Methods
  5. Results
  6. Experimental Setup
  7. Discussion
  8. Future Work

Figures (3)

  • Figure 1: Linear Frame Sampling vs Self-Reflective Sampling. Adapted from Self-ReS. In surveillance, events often occur in a non-linear timeline, and occupy a small spatial-temporal region of the video.
  • Figure 2: Self-Reflective Sampling. Adapted from Self-ReS. Self-ReS allows the LLM to self-reflectively sample the top most attended tokens by the LLM.
  • Figure 3: Confusion Matrix of VideoLLaMA2 VideoLLaMA2 results in UCF-Crime ucfcrime. The model performs well, but fails in situations where action definitions are very similar.