Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Differentiable Frequency-based Disentanglement for Aerial Video Action Recognition

Divya Kothandaraman, Ming Lin, Dinesh Manocha

TL;DR

The paper tackles action recognition in UAV videos where the human actor occupies a small portion of the frame and background motion confounds learning. It introduces a differentiable frequency-based framework that learns disentangled static and dynamic regions via a static-dynamic mask and an identity loss, and adds a parameter-free frame sampling strategy to select informative frames without retraining. The approach is designed to be embedded into any 3D CNN backbone, achieving state-of-the-art results on UAV Human and NEC Drone while maintaining lower computational overhead compared to transformer-based methods. Empirical results, including comprehensive ablations, demonstrate significant improvements and highlight the practical potential for robust aerial video analysis under dynamic camera conditions and low-resolution targets.

Abstract

We present a learning algorithm for human activity recognition in videos. Our approach is designed for UAV videos, which are mainly acquired from obliquely placed dynamic cameras that contain a human actor along with background motion. Typically, the human actors occupy less than one-tenth of the spatial resolution. Our approach simultaneously harnesses the benefits of frequency domain representations, a classical analysis tool in signal processing, and data driven neural networks. We build a differentiable static-dynamic frequency mask prior to model the salient static and dynamic pixels in the video, crucial for the underlying task of action recognition. We use this differentiable mask prior to enable the neural network to intrinsically learn disentangled feature representations via an identity loss function. Our formulation empowers the network to inherently compute disentangled salient features within its layers. Further, we propose a cost-function encapsulating temporal relevance and spatial content to sample the most important frame within uniformly spaced video segments. We conduct extensive experiments on the UAV Human dataset and the NEC Drone dataset and demonstrate relative improvements of 5.72% - 13.00% over the state-of-the-art and 14.28% - 38.05% over the corresponding baseline model.

Differentiable Frequency-based Disentanglement for Aerial Video Action Recognition

TL;DR

The paper tackles action recognition in UAV videos where the human actor occupies a small portion of the frame and background motion confounds learning. It introduces a differentiable frequency-based framework that learns disentangled static and dynamic regions via a static-dynamic mask and an identity loss, and adds a parameter-free frame sampling strategy to select informative frames without retraining. The approach is designed to be embedded into any 3D CNN backbone, achieving state-of-the-art results on UAV Human and NEC Drone while maintaining lower computational overhead compared to transformer-based methods. Empirical results, including comprehensive ablations, demonstrate significant improvements and highlight the practical potential for robust aerial video analysis under dynamic camera conditions and low-resolution targets.

Abstract

We present a learning algorithm for human activity recognition in videos. Our approach is designed for UAV videos, which are mainly acquired from obliquely placed dynamic cameras that contain a human actor along with background motion. Typically, the human actors occupy less than one-tenth of the spatial resolution. Our approach simultaneously harnesses the benefits of frequency domain representations, a classical analysis tool in signal processing, and data driven neural networks. We build a differentiable static-dynamic frequency mask prior to model the salient static and dynamic pixels in the video, crucial for the underlying task of action recognition. We use this differentiable mask prior to enable the neural network to intrinsically learn disentangled feature representations via an identity loss function. Our formulation empowers the network to inherently compute disentangled salient features within its layers. Further, we propose a cost-function encapsulating temporal relevance and spatial content to sample the most important frame within uniformly spaced video segments. We conduct extensive experiments on the UAV Human dataset and the NEC Drone dataset and demonstrate relative improvements of 5.72% - 13.00% over the state-of-the-art and 14.28% - 38.05% over the corresponding baseline model.
Paper Structure (27 sections, 5 equations, 2 figures, 6 tables)

This paper contains 27 sections, 5 equations, 2 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Main Contributions
  3. Related Work
  4. Aerial Video Recognition.
  5. Differentiability and Deep Learning.
  6. Motion Representation Methods.
  7. Frame sampling.
  8. Method
  9. Background: Differentiability
  10. Differentiable Static-Dynamic Frequency Mask
  11. Differentiable Disentanglement
  12. Frame Sampling
  13. Ensembling the predictions.
  14. Experiments
  15. Network and Training Details
  16. ...and 12 more sections

Figures (2)

  • Figure 1: Human activity recognition in videos recorded using dynamic cameras at oblique and aerial angles typically consist of human actors that occupy less than $10\%$ of the spatial resolution of the video. Background motion, noise, and low-resolution activities add to the complexities faced by video recognition systems. Our method achieves upto $41.86\%$ and $80.53\%$ and top-1 accuracies on UAV Human and NEC Drone, respectively.
  • Figure 2: We present a differentiable frequency-based method for aerial video recognition. Our differentiable static-dynamic frequency mask provides a prior for disentangled regions relevant to action recognition. This mask is used to guide the learning of disentangled features within the layers of the neural network using an identity function. Further, we propose a frame sampling strategy that chooses the best frame within each uniform video segment, at test time, using the static-dynamic frequency mask and temporal difference.