DINOv2 Rocks Geological Image Analysis: Classification, Segmentation, and Interpretability

TL;DR

A LoRA fine-tuned DINOv2, in particular, excels in out-of-distribution segmentation and outperforms other methods in multi-class tasks, even with limited data.

Abstract

Recent advancements in computer vision have significantly improved image analysis tasks. Yet, deep learning models often struggle when applied to domains outside their training distribution, such as in geosciences, where domain-specific data can be scarce. This study investigates the classification, segmentation, and interpretability of CT-scan images of rock samples, focusing on the application of modern computer vision techniques to geoscientific tasks. We compare a range of segmentation methods to assess their efficacy, efficiency, and adaptability in geological image analysis. The methods evaluated include Otsu thresholding, clustering techniques (K-means, fuzzy C-means), a supervised machine learning approach (Random Forest), and deep learning models (UNet, ResNet152, and DINOv2), using ten binary sandstone datasets and three multi-class calcite datasets. DINOv2 was selected for its promising results in feature extraction and its potential applicability in geoscientific tasks, prompting further assessment of its interpretability and effectiveness in processing CT-scanned rock data. For classification, a non-fine-tuned DINOv2 demonstrates strong performance in classifying rock images, even when the CT-scans are outside its original training set. In segmentation tasks, thresholding and clustering techniques, though computationally efficient, produce subpar results despite preprocessing efforts. In contrast, supervised methods achieve better performance. While deep learning methods demand greater computational resources, they require minimal intervention and offer superior generalization. A LoRA fine-tuned DINOv2, in particular, excels in out-of-distribution segmentation and outperforms other methods in multi-class tasks, even with limited data. Notably, the segmentation masks generated by DINOv2 often appear more accurate than the original targets, based on visual inspection.

Figures (12)

• Figure 1: Simplified illustration of DINOv2's ViT. Depending on the model size, the number of layers $L$ and the feature size $k$ may vary. We omit positional encoding for simplicity.
• Figure 2: t-SNE visualization of the sandstones dataset embedded in DINOv2 feature space.
• Figure 3: t-SNE visualization of pixel-level feature vectors for a test image from sample S3. This figure shows a 2-dimensional representation of DINOv2 and BFE clustering the pixel's latent representation of the same image.
• Figure 4: Confusion matrices for a segmentation prediction with kNN probing on a test image. Row indices of the confusion matrix correspond to the true class labels and column indices correspond to the predicted class labels.
• Figure 5: Visualization of the 3 principal components of CT-scanned images respectively using the raw features of DINOv2 (b) and using a fine-tuned linear head (c). The CT-scanner (a) and the GT (d) are displayed for easy comparison.