UPLiFT: Efficient Pixel-Dense Feature Upsampling with Local Attenders

TL;DR

UPLiFT addresses the challenge of producing dense pixel-dense features from pre-trained visual backbones without incurring the quadratic costs of cross-attention. It introduces a Local Attender, a locally constrained attention mechanism, enabling iterative 2x upsampling that scales linearly with the number of visual tokens. The approach achieves state-of-the-art results in semantic segmentation and depth estimation while maintaining faster inference than cross-attention–based upsamplers, and extends to generative tasks by upsampling VAE latent features with competitive quality and lower latency than Coupled Flow Matching. The work provides a versatile, efficient pathway for dense visual representations across predictive and generative tasks, with broad practical impact for dense vision systems.

Abstract

The space of task-agnostic feature upsampling has emerged as a promising area of research to efficiently create denser features from pre-trained visual backbones. These methods act as a shortcut to achieve dense features for a fraction of the cost by learning to map low-resolution features to high-resolution versions. While early works in this space used iterative upsampling approaches, more recent works have switched to cross-attention-based methods, which risk falling into the same efficiency scaling problems of the backbones they are upsampling. In this work, we demonstrate that iterative upsampling methods can still compete with cross-attention-based methods; moreover, they can achieve state-of-the-art performance with lower inference costs. We propose UPLiFT, an architecture for Universal Pixel-dense Lightweight Feature Transforms. We also propose an efficient Local Attender operator to overcome the limitations of prior iterative feature upsampling methods. This operator uses an alternative attentional pooling formulation defined fully locally. We show that our Local Attender allows UPLiFT to maintain stable features throughout upsampling, enabling state-of-the-art performance with lower inference costs than existing pixel-dense feature upsamplers. In addition, we apply UPLiFT to generative downstream tasks and show that it achieves competitive performance with state-of-the-art Coupled Flow Matching models for VAE feature upsampling. Altogether, UPLiFT offers a versatile and efficient approach to creating denser features.

Figures (15)

• Figure 1: UPLiFT time-scaling and dense features. We present UPLiFT, an efficient feature-upsampler that leverages our new Local Attender to extract semantically-stable, pixel-dense features. (Top) UPLiFT's inference time and memory scales linearly with the number of visual tokens, while other recent SOTA methods face quadratic scaling. (Bottom) PCA visualization of low-resolution DINOv2 features and pixel-dense UPLiFT features.
• Figure 2: UPLiFT Tasks. We demonstrate our UPLiFT feature upsampler for applications in both predictive and generative tasks. This includes semantic segmentation, monocular depth estimation, image super-resolution, and efficient text-to-image generation.
• Figure 3: UPLiFT Inference. At inference time, our UPLiFT Encoder ($\mathbf{E}_{\text{UPLiFT}}$) produces shallow but dense features to guide all subsequent upsampling steps. Iterative application of the UPLiFT Decoder ($\mathbf{D}_{\text{UPLiFT}}$) upsamples the low-resolution backbone features to pixel-density. Our proposed Local Attender module is integrated with the UPLiFT Decoder to maintain iterative feature consistency.
• Figure 4: Local Attender Operator. We propose a streamlined and efficient local attention operator, which gathers features over a set neighborhood defined by fixed direction offsets.
• Figure 5: UPLiFT Training. UPLiFT uses a multi-step training strategy where the feature reconstruction loss is applied at all intermediate steps. All lift decoders ($\mathbf{D}_{\text{UPLiFT}}$) share the same weights.