DiveUp: Learning Feature Upsampling from Diverse Vision Foundation Models

Abstract

Recently, feature upsampling has gained increasing attention owing to its effectiveness in enhancing vision foundation models (VFMs) for pixel-level understanding tasks. Existing methods typically rely on high-resolution features from the same foundation model to achieve upsampling via self-reconstruction. However, relying solely on intra-model features forces the upsampler to overfit to the source model's inherent location misalignment and high-norm artifacts. To address this fundamental limitation, we propose DiveUp, a novel framework that breaks away from single-model dependency by introducing multi-VFM relational guidance. Instead of naive feature fusion, DiveUp leverages diverse VFMs as a panel of experts, utilizing their structural consensus to regularize the upsampler's learning process, effectively preventing the propagation of inaccurate spatial structures from the source model. To reconcile the unaligned feature spaces across different VFMs, we propose a universal relational feature representation, formulated as a local center-of-mass (COM) field, that extracts intrinsic geometric structures, enabling seamless cross-model interaction. Furthermore, we introduce a spikiness-aware selection strategy that evaluates the spatial reliability of each VFM, effectively filtering out high-norm artifacts to aggregate guidance from only the most reliable expert at each local region. DiveUp is a unified, encoder-agnostic framework; a jointly-trained model can universally upsample features from diverse VFMs without requiring per-model retraining. Extensive experiments demonstrate that DiveUp achieves state-of-the-art performance across various downstream dense prediction tasks, validating the efficacy of multi-expert relational guidance. Our code and models are available at: https://github.com/Xiaoqiong-Liu/DiveUp

Figures (7)

• Figure 1: Overview of the DiveUp, which leverages diverse VFMs to learn feature upsampling. Best viewed in color and by zooming in for all the figures in this paper.
• Figure 2: The DiveUp Multi-VFM Relational Guidance Framework. (1) Relational Feature Representation. We compute local self-affinity (a) to evaluate spatial reliability via entropy (b) and extract the local center-of-mass (COM) field (c). In homogeneous regions, the expected spatial offset is negligible ($\boldsymbol{\mu}(p) \approx \mathbf{0}$), whereas it shifts directionally at semantic boundaries ($\boldsymbol{\mu}(p) \neq \mathbf{0}$), effectively capturing intrinsic geometric structures. (2) Spikiness-Aware COM Field Fusion. DiveUp extracts these relational representations from diverse VFMs (d) and employs a spikiness-aware source selection(e). By adaptively evaluating the spatial reliability of each VFM, we generate dynamic gating weights to fuse multiple COM fields into a unified Consensus COM Field(f). This serves as the robust location alignment target for high-resolution feature reconstruction.
• Figure 3: Comparison of upsampled features from different methods.
• Figure 4: Comparison of prediction results on semantic segmentation (a) in Pascal VOC and depth estimation (b) for different methods in NYUv2.
• Figure 5: Visualization of Relational Representations (Before vs. After DiveUp). We visualize the intermediate local entropy maps and COM fields generated by a noisy backbone (ClearClip-B), compared with those generated after integrating DiveUp's multi-VFM relational guidance. DiveUp suppresses background interference and spatial uncertainty, achieving more localized geometric structures. This improved location alignment corresponds to the quality of the final segmentation results (see IoU gains).