Multi-Scale Representations by Varying Window Attention for Semantic Segmentation

TL;DR

Semantic segmentation benefits from robust multi-scale representations, but existing methods exhibit scale inadequacy and field inactivation in their receptive fields. The authors introduce varying window attention (VWA) that disentangles the query and context to realize variable receptive fields, paired with a pre-scaling strategy (DOPE) and copy-shift padding (CSP) to maintain cost at parity with local window attention. A multi-scale decoder, VWFormer, built on VWA, fuses multi-scale representations through Multi-Layer Aggregation and enhances low-level features, achieving higher accuracy with similar or lower compute than strong baselines across ADE20K, Cityscapes, and COCOStuff-164K. Through ERF visualizations and extensive ablations, the approach demonstrates that all scales contribute and CSP mitigates attention collapse, delivering practical improvements for efficient semantic segmentation.

Abstract

Multi-scale learning is central to semantic segmentation. We visualize the effective receptive field (ERF) of canonical multi-scale representations and point out two risks in learning them: scale inadequacy and field inactivation. A novel multi-scale learner, varying window attention (VWA), is presented to address these issues. VWA leverages the local window attention (LWA) and disentangles LWA into the query window and context window, allowing the context's scale to vary for the query to learn representations at multiple scales. However, varying the context to large-scale windows (enlarging ratio R) can significantly increase the memory footprint and computation cost (R^2 times larger than LWA). We propose a simple but professional re-scaling strategy to zero the extra induced cost without compromising performance. Consequently, VWA uses the same cost as LWA to overcome the receptive limitation of the local window. Furthermore, depending on VWA and employing various MLPs, we introduce a multi-scale decoder (MSD), VWFormer, to improve multi-scale representations for semantic segmentation. VWFormer achieves efficiency competitive with the most compute-friendly MSDs, like FPN and MLP decoder, but performs much better than any MSDs. For instance, using nearly half of UPerNet's computation, VWFormer outperforms it by 1.0%-2.5% mIoU on ADE20K. With little extra overhead, ~10G FLOPs, Mask2Former armed with VWFormer improves by 1.0%-1.3%. The code and models are available at https://github.com/yan-hao-tian/vw

Figures (9)

• Figure 1: ERFs of multi-scale representations learned by (a) ASPP, (b) PSP, (c) ConvNeXt, (d) Swin Transformer, (e) SegFormer, and (f) Our proposed varying window attention. ERF maps are visualized across 100 images of ADE20K validation set. See Appendix \ref{['app:erf']} for more detailed analysis.
• Figure 2: (a) illustrates that in LWA, $\rm{\textbf{Q}}$, $\rm{\textbf{K}}$, and $\rm{\textbf{V}}$ are all transformed from the local window. (b) illustrates a naive implementation of VWA. $\rm{\textbf{Q}}$ is transformed from the local window. $\rm{\textbf{K}}$ and $\rm{\textbf{V}}$ are re-scaled from the varing window. PE is short for Patch Embedding. R (of RP) denotes the size ratio of the context window to the local window (query). (c) illustrates the professional implementation of VWA. DOPE is short for densely-overlapping Patch Embedding.
• Figure 3: (a) illustrates the zero-padding mode caused attention collapse when the context window is very large and the context window surrounds the local window near the corner or edge. (b) illustrates the proposed copy-shift padding (CSP) mode. The color change indicates where the padding pixels are from. (c) CSP is equivalent to moving the context windows towards the feature, ensuring that every pixel the query attends to has a different valid non-zero value. Best viewed in color.
• Figure 4: VWFormer contains multi-layer aggregation, learning multi-scale representations, and low-level enhancement. Like other MSDs, VWFormer takes multi-level feature maps as inputs.
• Figure 5: Visualization of inference results and ERFs of SegFormer and VWFormer. The red dot is the query location. The red box exhibits our method's receptive superiority. Zoom in to see details.