AnimalMotionCLIP: Embedding motion in CLIP for Animal Behavior Analysis

TL;DR

AnimalMotionCLIP tackles animal behavior recognition by embedding motion within the CLIP framework. It interleaves RGB frames with optical-flow fields and employs multiple shared-weight XCLIP classifiers with dense, semi-dense, and sparse temporal sampling, followed by score aggregation. The Video Encoder and Video-Guided Text Encoder enable dynamic video-text alignment, while ablations show sparse sampling and multimodal input yield the best performance. On the Animal Kingdom dataset, the approach surpasses state-of-the-art methods, demonstrating strong generalization to fine temporal actions without extensive pre-training, with practical implications for wildlife monitoring and animal welfare applications.

Abstract

Recently, there has been a surge of interest in applying deep learning techniques to animal behavior recognition, particularly leveraging pre-trained visual language models, such as CLIP, due to their remarkable generalization capacity across various downstream tasks. However, adapting these models to the specific domain of animal behavior recognition presents two significant challenges: integrating motion information and devising an effective temporal modeling scheme. In this paper, we propose AnimalMotionCLIP to address these challenges by interleaving video frames and optical flow information in the CLIP framework. Additionally, several temporal modeling schemes using an aggregation of classifiers are proposed and compared: dense, semi dense, and sparse. As a result, fine temporal actions can be correctly recognized, which is of vital importance in animal behavior analysis. Experiments on the Animal Kingdom dataset demonstrate that AnimalMotionCLIP achieves superior performance compared to state-of-the-art approaches.

Figures (3)

• Figure 1: Overview of the AnimalMotionCLIP system with an illustrative example of the frame selection process. The system processes color frames and motion fields from a video sequence, using a combination of main and contextual windows. Video recognition is performed by multiple XCLIP classifiers with shared weights, followed by score aggregation to generate the final prediction. The details are explained in \ref{['sec:method']}.
• Figure 2: Overview of the architecture of the classifiers in the Video Recognition module. It comprises a Video Encoder (left) and a Video Guided Text Encoder (right).
• Figure 3: Example of the Animal Kingdom dataset with different animal species in different environments and exhibiting diverse behaviors.