A Refreshed Similarity-based Upsampler for Direct High-Ratio Feature Upsampling

TL;DR

This work revisits similarity-based feature upsampling and identifies three key limitations: misalignment between high-resolution guidance and low-resolution features, rigid inner-product similarity, and mosaic artifacts from coarse neighbor selection. It introduces ReSFU, a refreshed framework that achieves explicit query-key alignment (semantic-aware mutual-alignment and detail-aware self-alignment), a learnable similarity calculation via Paired Central Difference Convolution, and fine-grained neighbor selection on HR features, enabling robust direct high-ratio upsampling across diverse architectures. Extensive experiments across semantic, instance, and panoptic segmentation, object detection, and monocular depth estimation demonstrate that ReSFU consistently outperforms baselines, often yielding sharper boundaries and fewer artifacts, with strong generality and deployment ease. The proposed approach offers practical impact by enabling high-quality, architecture-agnostic upsampling in dense prediction pipelines, reducing reliance on iterative guidance and enabling simpler, more versatile model designs.

Abstract

Feature upsampling is a fundamental and indispensable ingredient of almost all current network structures for dense prediction tasks. Recently, a popular similarity-based feature upsampling pipeline has been proposed, which utilizes a high-resolution feature as guidance to help upsample the low-resolution deep feature based on their local similarity. Albeit achieving promising performance, this pipeline has specific limitations: 1) HR query and LR key features are not well aligned; 2) the similarity between query-key features is computed based on the fixed inner product form; 3) neighbor selection is coarsely operated on LR features, resulting in mosaic artifacts. These shortcomings make the existing methods along this pipeline primarily applicable to hierarchical network architectures with iterative features as guidance and they are not readily extended to a broader range of structures, especially for a direct high-ratio upsampling. Against the issues, we meticulously optimize every methodological design. Specifically, we firstly propose an explicitly controllable query-key feature alignment from both semantic-aware and detail-aware perspectives, and then construct a parameterized paired central difference convolution block for flexibly calculating the similarity between the well-aligned query-key features. Besides, we develop a fine-grained neighbor selection strategy on HR features, which is simple yet effective for alleviating mosaic artifacts. Based on these careful designs, we systematically construct a refreshed similarity-based feature upsampling framework named ReSFU. Extensive experiments substantiate that our proposed ReSFU is finely applicable to various types of architectures in a direct high-ratio upsampling manner, and consistently achieves satisfactory performance on different dense prediction applications, showing superior generality and ease of deployment.

Figures (19)

• Figure 1: Upper: For the $\times 8$ upsampling from $C4$ to $\widetilde{C4}$ in the hierarchical SegFormer xie2021segformer, (a) SAPA lu2022sapa iteratively performs three $\times 2$ upsampling processes with intermediate features as guidance; (b) our ReSFU obtains clearer object boundaries in a direct one-step $\times 8$ upsampling. Lower: For the $\times 4$ upsampling from $\bm{x}$ to $\tilde{\bm{x}}$ under the guidance of the shallow HR feature $\bm{y}$ in the non-hierarchical Segmenter strudel2021segmenter, SAPA with (c) the default iterative upsampling incurs mosaic artifacts, which is worse in the (d) direct upsampling, while (e) ReSFU can largely eliminate the artifacts with one-step direct upsampling. All the images are best viewed by zooming in on screen, especially to observe mosaic artifacts.
• Figure 2: Targeting every part in (a) SAPA, (b) our ReSFU proposes specific optimization designs, i.e., controllable query-key feature alignment from both semantic-aware and detail-aware perspectives, paired central difference convolution (PCDC)-based flexible similarity calculation between aligned query-key pairs, and fine-grained neighbor selection (FNS). Here the visualization is experimentally based on Segmenter-S for a $\times 4$ feature upsampling from $\bm{x}$ to $\tilde{\bm{x}}$. Best viewed with zoom-in.
• Figure 3: Upper: For the two regions marked by blue and green boxes, i.e., the roof and the wall of the house, they have similar features in the semantic space but significantly differ in the detail space. Lower: The overall concept of our two-pronged approach to construct the aligned query-key pair in the semantic space and detail space, respectively.
• Figure 4: Illustration of the PCDC operation in Eq. (\ref{['eqn:pcd']}) for any input channel $d$ and any output channel $l$ where the bias term $\bm{b}_{l}$ is omitted for brevity. Here $\otimes$ is the convolution operation and $\ominus$ is the element-wise subtraction operation.
• Figure 5: Illustration of PCDC-Block for similarity calculation between the aligned query-key pair.