Fine-Tuning Video-Text Contrastive Model for Primate Behavior Retrieval from Unlabeled Raw Videos

TL;DR

This work tackles the challenge of retrieving capuchin monkey behaviors from unlabeled, noisy video footage collected in the wild. It introduces an agentic data treatment pipeline that leverages Multimodal Large Language Models to produce semantically aligned clip-text pairs, combined with Low-Rank Adaptation (LoRA) to fine-tune a pre-trained X-CLIP model for domain-specific retrieval and zero-shot classification. The approach yields substantial gains over raw pre-trained baselines on Hits@K and NDCG@K metrics, demonstrating the feasibility of domain adaptation from unlabeled data for wildlife behavior analysis. The results suggest a scalable path for analyzing large volumes of unlabeled ecological video data and motivate future work on larger datasets, training from scratch, and more capable, reasoning-enabled data-processing agents.

Abstract

Video recordings of nonhuman primates in their natural habitat are a common source for studying their behavior in the wild. We fine-tune pre-trained video-text foundational models for the specific domain of capuchin monkeys, with the goal of developing useful computational models to help researchers to retrieve useful clips from videos. We focus on the challenging problem of training a model based solely on raw, unlabeled video footage, using weak audio descriptions sometimes provided by field collaborators. We leverage recent advances in Multimodal Large Language Models (MLLMs) and Vision-Language Models (VLMs) to address the extremely noisy nature of both video and audio content. Specifically, we propose a two-folded approach: an agentic data treatment pipeline and a fine-tuning process. The data processing pipeline automatically extracts clean and semantically aligned video-text pairs from the raw videos, which are subsequently used to fine-tune a pre-trained Microsoft's X-CLIP model through Low-Rank Adaptation (LoRA). We obtained an uplift in $Hits@5$ of $167\%$ for the 16 frames model and an uplift of $114\%$ for the 8 frame model on our domain data. Moreover, based on $NDCG@K$ results, our model is able to rank well most of the considered behaviors, while the tested raw pre-trained models are not able to rank them at all. The code will be made available upon acceptance.

Figures (8)

• Figure 1: Video-Text Dual Encoder Architecture.
• Figure 1: Effect of LoRA rank and model layers in the retrieval ($Hits@K$) metrics.
• Figure 2: The figure presents two examples of video clips. In (a), we illustrate a commonly encountered video clip where the transcript poorly matches the content and the video itself is highly noisy. In (b), we display a video clip obtained after applying the proposed agentic data processing pipeline, resulting in significantly improved video quality and accurate alignment with its transcript. Copyright © LEDIS-USP archive.
• Figure 2: Effect of LoRA rank and model layers in Zero-shot accuracy.
• Figure 3: Data Generation Pipeline. In (a) one can see that OpenAI's Whisper is used to extract raw transcripts, which are then treated by a data processing Agent. The clean (clip, transcript) pairs are then submitted into BLIP-2 model (b) and only the pairs with cosine similarity greater than a predefined threshold are maintained, reducing the amount of noisy pairs. The diagram in (c) shows the actual graph produced with LangGraph. The graph is applied to each raw transcript individually.