Zero-Shot Wildlife Sorting Using Vision Transformers: Evaluating Clustering and Continuous Similarity Ordering

TL;DR

This paper investigates zero-shot organization of wildlife images from camera traps using self-supervised vision transformers, comparing discrete clustering with continuous 1D similarity ordering. Through a two-stage pipeline combining domain detectors and ViT features, the authors evaluate three ViT backbones with DBSCAN and GMM across PCA/UMAP, selecting GMM components via the Bayesian Information Criterion. The best clustering result (DINOv2 ViT-G/14 + UMAP + GMM) achieves 88.6% accuracy (macro-F1 0.874) on a 5-species test set, while continuous 1D ordering via t-SNE yields high coherence (88.2% for mammals/birds, 95.2% for fishes) across 1,500 images, highlighting preserved fine-grained morphology beyond discrete labels. The continuous ordering is deployed in production to accelerate exploratory analysis and annotation workflows for biodiversity monitoring, though scalability to larger species sets and the handling of difficult edge cases remain active areas for improvement.

Abstract

Camera traps generate millions of wildlife images, yet many datasets contain species that are absent from existing classifiers. This work evaluates zero-shot approaches for organizing unlabeled wildlife imagery using self-supervised vision transformers, developed and tested within the Animal Detect platform for camera trap analysis. We compare unsupervised clustering methods (DBSCAN, GMM) across three architectures (CLIP, DINOv2, MegaDescriptor) combined with dimensionality reduction techniques (PCA, UMAP), and we demonstrate continuous 1D similarity ordering via t-SNE projection. On a 5-species test set with ground truth labels used only for evaluation, DINOv2 with UMAP and GMM achieves 88.6 percent accuracy (macro-F1 = 0.874), while 1D sorting reaches 88.2 percent coherence for mammals and birds and 95.2 percent for fish across 1,500 images. Based on these findings, we deployed continuous similarity ordering in production, enabling rapid exploratory analysis and accelerating manual annotation workflows for biodiversity monitoring.