Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

ChimpVLM: Ethogram-Enhanced Chimpanzee Behaviour Recognition

Otto Brookes, Majid Mirmehdi, Hjalmar Kuhl, Tilo Burghardt

TL;DR

This work tackles automatic chimpanzee behaviour recognition from camera-trap videos, addressing long-tail class distributions and semantic grounding. It proposes ChimpVLM, a vision-language model that initialises per-behaviour query tokens from an ethogram via a fine-tuned language model and decodes them with a multi-modal decoder that fuses spatio-temporal video features. The approach achieves state-of-the-art results on PanAf500 (top-1) and PanAf20K (overall and tail mAP), with the strongest gains when using full behavioural descriptions and in-domain LM fine-tuning. The results demonstrate the value of integrating ethogram-driven textual priors with visual representations for ecological monitoring, and the authors release code and weights to support reproducibility and reuse.

Abstract

We show that chimpanzee behaviour understanding from camera traps can be enhanced by providing visual architectures with access to an embedding of text descriptions that detail species behaviours. In particular, we present a vision-language model which employs multi-modal decoding of visual features extracted directly from camera trap videos to process query tokens representing behaviours and output class predictions. Query tokens are initialised using a standardised ethogram of chimpanzee behaviour, rather than using random or name-based initialisations. In addition, the effect of initialising query tokens using a masked language model fine-tuned on a text corpus of known behavioural patterns is explored. We evaluate our system on the PanAf500 and PanAf20K datasets and demonstrate the performance benefits of our multi-modal decoding approach and query initialisation strategy on multi-class and multi-label recognition tasks, respectively. Results and ablations corroborate performance improvements. We achieve state-of-the-art performance over vision and vision-language models in top-1 accuracy (+6.34%) on PanAf500 and overall (+1.1%) and tail-class (+2.26%) mean average precision on PanAf20K. We share complete source code and network weights for full reproducibility of results and easy utilisation.

ChimpVLM: Ethogram-Enhanced Chimpanzee Behaviour Recognition

TL;DR

This work tackles automatic chimpanzee behaviour recognition from camera-trap videos, addressing long-tail class distributions and semantic grounding. It proposes ChimpVLM, a vision-language model that initialises per-behaviour query tokens from an ethogram via a fine-tuned language model and decodes them with a multi-modal decoder that fuses spatio-temporal video features. The approach achieves state-of-the-art results on PanAf500 (top-1) and PanAf20K (overall and tail mAP), with the strongest gains when using full behavioural descriptions and in-domain LM fine-tuning. The results demonstrate the value of integrating ethogram-driven textual priors with visual representations for ecological monitoring, and the authors release code and weights to support reproducibility and reuse.

Abstract

We show that chimpanzee behaviour understanding from camera traps can be enhanced by providing visual architectures with access to an embedding of text descriptions that detail species behaviours. In particular, we present a vision-language model which employs multi-modal decoding of visual features extracted directly from camera trap videos to process query tokens representing behaviours and output class predictions. Query tokens are initialised using a standardised ethogram of chimpanzee behaviour, rather than using random or name-based initialisations. In addition, the effect of initialising query tokens using a masked language model fine-tuned on a text corpus of known behavioural patterns is explored. We evaluate our system on the PanAf500 and PanAf20K datasets and demonstrate the performance benefits of our multi-modal decoding approach and query initialisation strategy on multi-class and multi-label recognition tasks, respectively. Results and ablations corroborate performance improvements. We achieve state-of-the-art performance over vision and vision-language models in top-1 accuracy (+6.34%) on PanAf500 and overall (+1.1%) and tail-class (+2.26%) mean average precision on PanAf20K. We share complete source code and network weights for full reproducibility of results and easy utilisation.
Paper Structure (8 sections, 1 equation, 4 figures, 2 tables)

This paper contains 8 sections, 1 equation, 4 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Method
  4. Experiments and Results
  5. Setup
  6. PanAf500
  7. PanAf20k
  8. Conclusion

Figures (4)

  • Figure 1: Camera Reaction Ethogram. A series of still frames (left) demonstrates the intra-class variation of the camera reactivity behaviour. The ethogram (right) describes the behavioural patterns corresponding to each of the different variations and provides a semantically richer representation of the behaviour.
  • Figure 2: System Overview. Our proposed system employs a standard vision encoder, a fine-tuned large language model, and a multi-model decoder comprising several cross-attention layers. Query tokens are initialised by embedding either behaviour names or descriptions using a large language model. During training, the multi-modal decoder utilises spatio-temporal visual features to decode query tokens and output logits for downstream classification.
  • Figure 3: PanAf500 Class-wise Performance vs. Proportion of Data. The per-class accuracy of ChimpVLM (CLS+FT) and Internvideo is plotted against the proportion of data for each class in the PanAf500 dataset.
  • Figure 4: PanAf20K Class-wise Performance vs. Proportion of Data. The per-class AP of ChimpVLM (DSC+FT) and Internvideo is plotted against the proportion of data for each of the bottom 9 classes in the PanAf20k dataset.