LDA-AQU: Adaptive Query-guided Upsampling via Local Deformable Attention

TL;DR

Upsampling in convolutional networks often struggles to provide task-specific guidance without high-resolution inputs. The paper introduces LDA-AQU, a lightweight, single-layer upsampling operator that uses local deformable attention guided by query features to adapt both the neighbor positions and aggregation weights. This framework unifies many existing upsampling schemes under a query-guided, dynamic mechanism and delivers consistent improvements across object detection, instance segmentation, panoptic segmentation, and semantic segmentation on MS COCO and ADE20K with minimal increases in FLOPs and parameters. The approach demonstrates that query-guided deformable local attention is a powerful primitive for dense prediction tasks, enabling more accurate feature reconstruction and better multi-scale fusion in vision models.

Abstract

Feature upsampling is an essential operation in constructing deep convolutional neural networks. However, existing upsamplers either lack specific feature guidance or necessitate the utilization of high-resolution feature maps, resulting in a loss of performance and flexibility. In this paper, we find that the local self-attention naturally has the feature guidance capability, and its computational paradigm aligns closely with the essence of feature upsampling (\ie feature reassembly of neighboring points). Therefore, we introduce local self-attention into the upsampling task and demonstrate that the majority of existing upsamplers can be regarded as special cases of upsamplers based on local self-attention. Considering the potential semantic gap between upsampled points and their neighboring points, we further introduce the deformation mechanism into the upsampler based on local self-attention, thereby proposing LDA-AQU. As a novel dynamic kernel-based upsampler, LDA-AQU utilizes the feature of queries to guide the model in adaptively adjusting the position and aggregation weight of neighboring points, thereby meeting the upsampling requirements across various complex scenarios. In addition, LDA-AQU is lightweight and can be easily integrated into various model architectures. We evaluate the effectiveness of LDA-AQU across four dense prediction tasks: object detection, instance segmentation, panoptic segmentation, and semantic segmentation. LDA-AQU consistently outperforms previous state-of-the-art upsamplers, achieving performance enhancements of 1.7 AP, 1.5 AP, 2.0 PQ, and 2.5 mIoU compared to the baseline models in the aforementioned four tasks, respectively. Code is available at \url{https://github.com/duzw9311/LDA-AQU}.

Figures (12)

• Figure 1: Comparison of various upsamplers in terms of network parameters, Mean Average Precision (mAP) and FLOPs (indicated by area of circles) using Faster R-CNN ren2015faster with ResNet-50 he2016deep as the baseline model.
• Figure 2: The difference in neighboring point selection schemes between LDA-AQU and other upsamplers including Nearest Neighbor Interpolation, Bilinear Interpolation, and CARAFE wang2019carafe. Given an upsampled point (red star), LDA-AQU employs the query-guided mechanism to predict the deformation offset and aggregation weight of neighboring points, enabling adaptation to upsampling tasks across multiple scales.
• Figure 3: The upsampling kernels of Bilinear Interpolation-Based (a) and Local Self-Attention-Based Upsamplers (b).
• Figure 4: The overall framework of LDA-AQU. Given an input feature map with size $H \times W \times C$ and an upsampling factor $\alpha$, LDA-AQU employs local deformable attention for feature upsampling, resulting in an output feature map with size $\alpha H \times \alpha W \times C$.
• Figure 5: Visualizations of some upsampled points (first row) and their deformed neighboring points (second row). Colored rings depict upsampled points, while scatter points of the same color are corresponding deformed neighboring points.