Vision Transformers for Zero-Shot Clustering of Animal Images: A Comparative Benchmarking Study

TL;DR

This work tackles the bottleneck of manually labeling vast ecological image collections by evaluating zero-shot clustering using Vision Transformer embeddings across 60 species. The authors benchmark five ViT models with 23 dimensionality-reduction configurations and 12 clustering setups, revealing that DINOv3 embeddings combined with 2D t-SNE or UMAP and HDBSCAN achieve near-perfect species-level clustering (V-measure up to $0.958$) while unsupervised approaches reach $0.943$ with minimal outliers. They also demonstrate ecologically meaningful intra-specific structure, such as age, sex, and pelage variation, emerging under intentional over-clustering. Their open-source toolkit, validated dataset, and deployment guidelines provide practical pathways for ecologists to sort large-scale biodiversity data while identifying edge cases requiring expert review. Overall, the study establishes a robust, scalable framework for zero-shot biodiversity clustering that can extend to other taxa and imaging contexts.

Abstract

Manual labeling of animal images remains a significant bottleneck in ecological research, limiting the scale and efficiency of biodiversity monitoring efforts. This study investigates whether state-of-the-art Vision Transformer (ViT) foundation models can reduce thousands of unlabeled animal images directly to species-level clusters. We present a comprehensive benchmarking framework evaluating five ViT models combined with five dimensionality reduction techniques and four clustering algorithms, two supervised and two unsupervised, across 60 species (30 mammals and 30 birds), with each test using a random subset of 200 validated images per species. We investigate when clustering succeeds at species-level, where it fails, and whether clustering within the species-level reveals ecologically meaningful patterns such as sex, age, or phenotypic variation. Our results demonstrate near-perfect species-level clustering (V-measure: 0.958) using DINOv3 embeddings with t-SNE and supervised hierarchical clustering methods. Unsupervised approaches achieve competitive performance (0.943) while requiring no prior species knowledge, rejecting only 1.14% of images as outliers requiring expert review. We further demonstrate robustness to realistic long-tailed distributions of species and show that intentional over-clustering can reliably extract intra-specific variation including age classes, sexual dimorphism, and pelage differences. We introduce an open-source benchmarking toolkit and provide recommendations for ecologists to select appropriate methods for sorting their specific taxonomic groups and data.

Figures (13)

• Figure 1: Left side; 100 randomly ordered images from 5 mammal classes. Right side; images sorted in 2-D embedding space clusters, from results of DINOV3, t-SNE and HDBSCAN.
• Figure 2: Data preparation pipeline using the Animal Detect platform. First step using the MegaDetector1000 Redwood model to detect animals, shown with red bounding boxes. Second step, cropping the images, using the bounding box location and third step with expert validation, ensuring the classifications are correct, re-labeling uncertain cases.
• Figure 3: Zero-shot clustering pipeline architecture. Each stage is modular, allowing substitution of different models and methods as new approaches emerge.
• Figure 4: Benchmarking framework architecture. The complete pipeline is repeated 10 times for mammals and 10 for birds, with different random image subsets to generalize beyond a single specific sample selection.
• Figure 5: V-measure distribution across vision transformer models. Box plots show median, interquartile range, and outliers for all 5,520 configurations per model.