Benchmarking Feature Upsampling Methods for Vision Foundation Models using Interactive Segmentation

TL;DR

This paper addresses the challenge of limited spatial resolution in Vision Foundation Models (VFMs) for dense prediction by evaluating task-agnostic, image-guided feature upsamplers within an Interactive Segmentation (IS) framework. The authors propose a streamlined benchmark where a frozen VFM is followed by a learnable upsampler and a lightweight segmentation head, enabling fair comparison across upsampling methods without retraining the backbone. Through extensive experiments on four IS datasets, LoftUp emerges as the most effective upsampler, leveraging full-resolution pseudo-GT supervision and a coordinate-based cross-attention design to achieve substantial performance gains over bilinear upsampling. The study validates IS as a structured, real-world benchmark for dense prediction with VFMs and highlights the practical impact of advanced upsampling modules in improving VFM feature quality and segmentation accuracy. $NoC$ and $IoU$-based metrics show that additional user clicks progressively reduce ambiguity, with LoftUp delivering robust improvements across datasets and click regimes.

Abstract

Vision Foundation Models (VFMs) are large-scale, pre-trained models that serve as general-purpose backbones for various computer vision tasks. As VFMs' popularity grows, there is an increasing interest in understanding their effectiveness for dense prediction tasks. However, VFMs typically produce low-resolution features, limiting their direct applicability in this context. One way to tackle this limitation is by employing a task-agnostic feature upsampling module that refines VFM features resolution. To assess the effectiveness of this approach, we investigate Interactive Segmentation (IS) as a novel benchmark for evaluating feature upsampling methods on VFMs. Due to its inherent multimodal input, consisting of an image and a set of user-defined clicks, as well as its dense mask output, IS creates a challenging environment that demands comprehensive visual scene understanding. Our benchmarking experiments show that selecting appropriate upsampling strategies significantly improves VFM features quality. The code is released at https://github.com/havrylovv/iSegProbe

Figures (8)

• Figure 2: Comparison of features from upsamplers with a single input click. The click is indicated by a green dot on the original images. Backbone is DINOv2 (S/14) DBLP:journals/tmlr/OquabDMVSKFHMEA24. Click encoder is a symmetric patch embedding with early injection. Visualization method follows the PCA-based approach introduced in FeatUp DBLP:conf/iclr/FuHBFZF24.
• Figure 3: Detailed example of IS inference. The model takes an input image along with a stacked representation of two disk maps indicating positive and negative clicks. Positive and negative clicks are indicated by green and red dots, respectively. Additionally, the model may receive a probability map from the previous iteration. The input image is not shown for clarity.
• Figure 4: Segmentation results on GrabCutDBLP:journals/tog/RotherKB04 with a single input click. Successful cases. The click is indicated by a green dot. Backbone is DINOv2 (S/14) DBLP:journals/tmlr/OquabDMVSKFHMEA24. Click encoder is a symmetric patch embedding with early injection.
• Figure 5: Segmentation results on GrabCutDBLP:journals/tog/RotherKB04 with a single input click. Failure cases. The click is indicated by a green dot. Backbone is DINOv2 (S/14) DBLP:journals/tmlr/OquabDMVSKFHMEA24. Click encoder is a symmetric patch embedding with early injection.
• Figure 6: Convergence of IoU with increasing user clicks. Backbone is DINOv2 (S/14) DBLP:journals/tmlr/OquabDMVSKFHMEA24. Click encoder is a symmetric patch embedding with early injection.