DeepAndes: A Self-Supervised Vision Foundation Model for Multi-Spectral Remote Sensing Imagery of the Andes

TL;DR

DeepAndes presents a transformer-based vision foundation model tailored to 8-band multispectral remote sensing for the Andes, trained on 3 million WorldView patches with self-supervised learning via a DINOv2 framework. It demonstrates strong few-shot and zero-shot performance on imbalanced archaeological tasks—imbalanced loci classification, image instance retrieval, and few-shot segmentation—exhibiting a scaling law where larger pre-training yields better representations. The work underscores the value of large-scale self-supervised pre-training for domain-specific remote sensing in archaeology and offers a path toward scalable, data-efficient analysis with potential broad applicability in geoscience. The model is integrated with downstream workflows and points toward future enhancements, including human-in-the-loop validation and domain-adapted extensions like DeepAndesArch.

Abstract

By mapping sites at large scales using remotely sensed data, archaeologists can generate unique insights into long-term demographic trends, inter-regional social networks, and past adaptations to climate change. Remote sensing surveys complement field-based approaches, and their reach can be especially great when combined with deep learning and computer vision techniques. However, conventional supervised deep learning methods face challenges in annotating fine-grained archaeological features at scale. While recent vision foundation models have shown remarkable success in learning large-scale remote sensing data with minimal annotations, most off-the-shelf solutions are designed for RGB images rather than multi-spectral satellite imagery, such as the 8-band data used in our study. In this paper, we introduce DeepAndes, a transformer-based vision foundation model trained on three million multi-spectral satellite images, specifically tailored for Andean archaeology. DeepAndes incorporates a customized DINOv2 self-supervised learning algorithm optimized for 8-band multi-spectral imagery, marking the first foundation model designed explicitly for the Andes region. We evaluate its image understanding performance through imbalanced image classification, image instance retrieval, and pixel-level semantic segmentation tasks. Our experiments show that DeepAndes achieves superior F1 scores, mean average precision, and Dice scores in few-shot learning scenarios, significantly outperforming models trained from scratch or pre-trained on smaller datasets. This underscores the effectiveness of large-scale self-supervised pre-training in archaeological remote sensing. Codes will be available on https://github.com/geopacha/DeepAndes.

Figures (8)

• Figure 1: Overview of DeepAndes. This figure shows the training dataset (a-d) and three domain-specific downstream tasks (e) using DeepAndes — a vision foundation model designed for multi-spectral satellite imagery in the Andes region. Particularly, (a) shows a large-scale map of the imagery used to train DeepAndes, highlighting various land cover types, with their area distribution shown in (c). (b) presents the unit sample patch (red box in a, b, d) with eight spectral bands. (d) illustrates image patching for DINOv2 training, with geospatial sampling densely covering different archaeological sites.
• Figure 2: Scaling law is observed in DeepAndes. This figure illustrates the model’s performance across three key downstream tasks: site classification, image retrieval, and building segmentation. The results are presented for models trained with no pretraining (w/o), 30K, 300K, and 3 million images. The findings highlight the scalability of DeepAndes, indicating that its performance can be further improved with larger training datasets.
• Figure 3: DINOv2: the self-supervised contrastive representation learning algorithm with knowledge distillation. (A) shows an overview of the framework. (B) illustrates the details of the DINOv2 multi-crop SSL training scheme.
• Figure 4: Image Understanding Downstream Analysis. Both image-level tasks (e.g., image classification and retrieval) and pixel-level tasks (e.g., segmentation) are included.
• Figure 5: Performance on Imbalanced Loci Classification: Precision-Recall curves from five-fold cross-validation (a). Confusion matrices (normalized) for each model shown for a representative hold-out fold (b).