P2AT: Pyramid Pooling Axial Transformer for Real-time Semantic Segmentation

TL;DR

P2AT presents a real-time semantic segmentation framework that hybrids CNNs and Transformers by incorporating a Pyramid Pooling Axial Transformer to capture global context while preserving efficiency. The architecture introduces a Bidirectional Fusion module, a Global Context Enhancer, and a lightweight decoder to fuse multi-level features and refine predictions, achieving strong performance on driving scenes. Empirical results show state-of-the-art mIoU on Camvid and competitive accuracy with efficient inference on Cityscapes and VOC2012, with P2AT-M offering the best accuracy–speed balance. This work provides a practical approach for accurate and fast dense prediction suitable for autonomous driving systems.

Abstract

Recently, Transformer-based models have achieved promising results in various vision tasks, due to their ability to model long-range dependencies. However, transformers are computationally expensive, which limits their applications in real-time tasks such as autonomous driving. In addition, an efficient local and global feature selection and fusion are vital for accurate dense prediction, especially driving scene understanding tasks. In this paper, we propose a real-time semantic segmentation architecture named Pyramid Pooling Axial Transformer (P2AT). The proposed P2AT takes a coarse feature from the CNN encoder to produce scale-aware contextual features, which are then combined with the multi-level feature aggregation scheme to produce enhanced contextual features. Specifically, we introduce a pyramid pooling axial transformer to capture intricate spatial and channel dependencies, leading to improved performance on semantic segmentation. Then, we design a Bidirectional Fusion module (BiF) to combine semantic information at different levels. Meanwhile, a Global Context Enhancer is introduced to compensate for the inadequacy of concatenating different semantic levels. Finally, a decoder block is proposed to help maintain a larger receptive field. We evaluate P2AT variants on three challenging scene-understanding datasets. In particular, our P2AT variants achieve state-of-art results on the Camvid dataset 80.5%, 81.0%, 81.1% for P2AT-S, P2ATM, and P2AT-L, respectively. Furthermore, our experiment on Cityscapes and Pascal VOC 2012 have demonstrated the efficiency of the proposed architecture, with results showing that P2AT-M, achieves 78.7% on Cityscapes. The source code will be available at

Figures (11)

• Figure 1: The inference speed and accuracy trade-off for real-time models on the Cityscapes cordts2016cityscapes test set. red color refers to our models, while black represents others.
• Figure 2: The architecture of P2AT. (a) Encoder based on pre-trained ResNet, (b) Transformer Layers to extract contextual information, (c) Multi-stage Feature Fusion Block (d) Decoder Block (e) Feature Refinement Block.
• Figure 3: Efficient Bidirectional fusion Module.
• Figure 4: Global Context Enhancer Module.
• Figure 5: Illustrates the detail of the (a) feature refinement module and (b) feature decoding block.