Reviving the Context: Camera Trap Species Classification as Link Prediction on Multimodal Knowledge Graphs

TL;DR

This work tackles the challenge of out-of-distribution generalization in camera-trap species classification by leveraging rich contextual information. It introduces COSMO, a framework that casts visual recognition as link prediction on a multimodal knowledge graph integrating images, taxonomy, location, and time, using a DistMult backbone with modality-specific encoders. The approach yields competitive results on iWildCam2020-WILDS and Snapshot Mountain Zebra, with clear gains from including contextual data, especially location, and demonstrates improved sample efficiency for under-represented species. The findings highlight the practical potential of multimodal knowledge graphs to enhance robustness and generalization in ecological monitoring tasks, while pointing to avenues for weighting context informativeness and expanding context types in future work.

Abstract

Camera traps are important tools in animal ecology for biodiversity monitoring and conservation. However, their practical application is limited by issues such as poor generalization to new and unseen locations. Images are typically associated with diverse forms of context, which may exist in different modalities. In this work, we exploit the structured context linked to camera trap images to boost out-of-distribution generalization for species classification tasks in camera traps. For instance, a picture of a wild animal could be linked to details about the time and place it was captured, as well as structured biological knowledge about the animal species. While often overlooked by existing studies, incorporating such context offers several potential benefits for better image understanding, such as addressing data scarcity and enhancing generalization. However, effectively incorporating such heterogeneous context into the visual domain is a challenging problem. To address this, we propose a novel framework that transforms species classification as link prediction in a multimodal knowledge graph (KG). This framework enables the seamless integration of diverse multimodal contexts for visual recognition. We apply this framework for out-of-distribution species classification on the iWildCam2020-WILDS and Snapshot Mountain Zebra datasets and achieve competitive performance with state-of-the-art approaches. Furthermore, our framework enhances sample efficiency for recognizing under-represented species.

Figures (5)

• Figure 1: Overview of our framework COSMO.Left: Our multimodal knowledge graph for camera traps and wildlife. Photos from camera traps are jointly represented in the KG with contextual information such as time, location, and structured biology taxonomy. The taxonomy is obtained from Open Tree Taxonomy (OTT) OTT or iNaturalist van2018inaturalist. Right: In our formulation of species classification as link prediction, the plausibility score $\psi(s, r, o)$ of each (subject, relation, object) triplet is computed using a KGE model (e.g., DistMult), where the subject, relation, and object are all first embedded into a vector space. Specifically, for our multimodal KG, we represent visual entities using a ResNet-50 pre-trained on ImageNet and represent numerical entities using an MLP. For categorical entities and relations, we directly represent them with embedding lookups.
• Figure 2: Comparison of COSMO model with and without taxonomy edges (iWildCam2020-WILDS validation set). The use of taxonomy information helps the model to avoid semantically implausible predictions.
• Figure 3: Correlation analysis for location and time attributes. Best viewed in color.
• Figure 4: Plot of location GPS coordinates for training and validation splits (iWildCam2020-WILDS). The coordinates can be grouped into six clusters. Most coordinates exhibit an overlap with their respective cluster centroids at this visualization scale. Best viewed in color.
• Figure C.1: Species probabilities conditioned on day/night for the 10 most frequent species in the training set (iWildCam2020-WILDS). Animal species demonstrate distinct temporal preferences for their daily activities, as evidenced by the contrasting probabilities observed during day and night.