AnyUp: Universal Feature Upsampling

TL;DR

AnyUp tackles the problem of upsampling features from any vision encoder to any resolution without encoder-specific training. It introduces a feature-agnostic layer, local window attention, and an image-part based training strategy to achieve high-quality, encoder-agnostic upsampling, validated across semantic segmentation, depth, and normal estimation tasks. The approach outperforms prior methods, preserves the original feature space, and generalizes to unseen feature types, enabling practical deployment across diverse models. This work delivers a practical, scalable solution for pixel-level vision tasks with broad applicability and a public codebase.

Abstract

We introduce AnyUp, a method for feature upsampling that can be applied to any vision feature at any resolution, without encoder-specific training. Existing learning-based upsamplers for features like DINO or CLIP need to be re-trained for every feature extractor and thus do not generalize to different feature types at inference time. In this work, we propose an inference-time feature-agnostic upsampling architecture to alleviate this limitation and improve upsampling quality. In our experiments, AnyUp sets a new state of the art for upsampled features, generalizes to different feature types, and preserves feature semantics while being efficient and easy to apply to a wide range of downstream tasks.

Figures (12)

• Figure 1: AnyUp is a universal feature upsampling model that can upsample any feature from any to any resolution, generalizing to unseen features while achieving state-of-the-art performance.
• Figure 2: Visual comparison against other methods. RGB channels correspond to the first three principal components computed over all features. Previous methods result in excessive smoothing or contain other artifacts: See, e.g., in the first row, the smoothed-out cloud features in LoftUp or the feature distribution shift for the mountains in JAFAR, as well as the oversmoothing and halo-artifacts of FeatUp and Guided Filter in the third row. AnyUp (Ours) produces sharp output feature maps while preserving the input feature quality.
• Figure 3: Method Overview. AnyUp performs window attention-based upsampling (\ref{['sec:window-attention']}). Input features are processed with a feature-agnostic layer (\ref{['sec:feature-agnostic-layer']}). During training, features computed for randomly sampled image parts are used as a reference for the respective part of the upsampled feature map (\ref{['sec:training-pipeline']}).
• Figure 4: Feature-agnostic layer. Input channels are processed independently and contributions to basis filters are averaged over all channels leading to outputs invariant to input dimensionality.
• Figure 5: Qualitative Probing Results. Visualization of linear probing results for monocular depth estimation on NYUv2 and semantic segmentation on ADE20k. More visualizations given in App. \ref{['sec:appendix-visualizations']}.