Embedding-Free Transformer with Inference Spatial Reduction for Efficient Semantic Segmentation

TL;DR

This work targets efficient semantic segmentation by introducing Embedding-Free Attention (EFA) within an EFT encoder and an all-attention decoder, forming the EDAFormer. A key novelty is Inference Spatial Reduction (ISR), which lowers key/value resolution during inference to cut computation with minimal accuracy loss. Empirically, EDAFormer delivers strong efficiency-accuracy performance on ADE20K, Cityscapes, and COCO-Stuff, with ISR capable of reducing FLOPs by up to ~61% on Cityscapes without substantial mIoU degradation and without fine-tuning. The approach also generalizes to other transformer backbones, offering a practical pathway to deploy efficient, global-context-aware transformers for semantic segmentation.

Abstract

We present an Encoder-Decoder Attention Transformer, EDAFormer, which consists of the Embedding-Free Transformer (EFT) encoder and the all-attention decoder leveraging our Embedding-Free Attention (EFA) structure. The proposed EFA is a novel global context modeling mechanism that focuses on functioning the global non-linearity, not the specific roles of the query, key and value. For the decoder, we explore the optimized structure for considering the globality, which can improve the semantic segmentation performance. In addition, we propose a novel Inference Spatial Reduction (ISR) method for the computational efficiency. Different from the previous spatial reduction attention methods, our ISR method further reduces the key-value resolution at the inference phase, which can mitigate the computation-performance trade-off gap for the efficient semantic segmentation. Our EDAFormer shows the state-of-the-art performance with the efficient computation compared to the existing transformer-based semantic segmentation models in three public benchmarks, including ADE20K, Cityscapes and COCO-Stuff. Furthermore, our ISR method reduces the computational cost by up to 61% with minimal mIoU performance degradation on Cityscapes dataset. The code is available at https://github.com/hyunwoo137/EDAFormer.

Figures (8)

• Figure 1: (a) Overall architecture of the proposed EDAFormer, consisting of two main parts: an EFT encoder and an all-attention decoder. The encoder and decoder of EDAFormer are designed with the query, key and value embedding free attention structure. (b) Details of the EFT block that contains EFA module.
• Figure 2: Comparison of the previous method and our EFA.
• Figure 3: Overview of our ISR method at the encoder stage-1. Our ISR applies the reduction ratio at the inference, reducing the key and value tokens selectively. This framework can be performed at every stage that contains the self-attention structure. It leads to flexibly reduce the computational cost without disrupting the spatial structure.
• Figure 4: Visualization of the attention score map, output features, and prediction map on ADE20K. 'Base' represents our EDAFormer trained with the base reduction ratio of [8,4,2,1]-[1,2,4]. 'w/ ISR' represents our EDAFormer applied our ISR method.
• Figure 5: Qualitative results on ADE20K, Cityscapes, and COCO-Stuff. Compared to SegFormer, the predictions of our EDAFormer are more precise for various categories.