Analyzing Local Representations of Self-supervised Vision Transformers

TL;DR

This paper examines the abilities of ViTs to perform various computer vision tasks with little to no fine-tuning, and designs evaluation framework to analyze the quality of local patch-level representations in the context of few-shot semantic segmentation, instance identification, object retrieval and tracking.

Abstract

In this paper, we present a comparative analysis of various self-supervised Vision Transformers (ViTs), focusing on their local representative power. Inspired by large language models, we examine the abilities of ViTs to perform various computer vision tasks with little to no fine-tuning. We design evaluation framework to analyze the quality of local, i.e.\ patch-level, representations in the context of few-shot semantic segmentation, instance identification, object retrieval and tracking. We discover that contrastive learning based methods like DINO produce more universal patch representations that can be immediately applied for downstream tasks with no parameter tuning, compared to masked image modeling. The embeddings learned using the latter approach, e.g. in masked autoencoders, have high variance features that harm distance-based algorithms, such as k-NN, and do not contain useful information for most downstream tasks. Furthermore, we demonstrate that removing these high-variance features enhances k-NN for MAE, as well as for its recent extension Scale-MAE. Finally, we find an object instance retrieval setting where DINOv2, a model pretrained on two orders of magnitude more data, falls short of its less compute intensive counterpart DINO.

Figures (13)

• Figure 1: mIoU of patch classifiers on various subsets of Cityscapes dataset. The evaluation is performed on the same validation set. The size of the circles are proportional to the number of images in the training set (36, 72, and 144).
• Figure 2: Analysis of high-variance features of ViT patch embeddings.
• Figure 3: k-NN patch classification performance (mIoU) for various vision transformers on few-shot subset of Cityscapes.
• Figure 4: Robustness of linear models for segmentation on Cityscapes. For every model (differing by colors) we have five dots, each representing one level of degradation. y-axis shows the mIoU on corrupted images. The top dots in vertical lines correspond to the original images. Grey diagonal lines indicate levels of equal relative drop in mIoU.
• Figure 5: Stacked area plots for each model (columns) and transformation type (rows). In each subplot, x-axis is the level of transformation, y-axis is the percentage of correctly matched patches in each granularity.