Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

ICFRNet: Image Complexity Prior Guided Feature Refinement for Real-time Semantic Segmentation

Xin Zhang, Teodor Boyadzhiev, Jinglei Shi, Jufeng Yang

TL;DR

The Image Complexity prior-guided Feature Refinement Network (ICFRNet) is proposed, which aggregates both complexity and segmentation features to produce an attention map for refining segmentation features within an Image Complexity Guided Attention (ICGA) module.

Abstract

In this paper, we leverage image complexity as a prior for refining segmentation features to achieve accurate real-time semantic segmentation. The design philosophy is based on the observation that different pixel regions within an image exhibit varying levels of complexity, with higher complexities posing a greater challenge for accurate segmentation. We thus introduce image complexity as prior guidance and propose the Image Complexity prior-guided Feature Refinement Network (ICFRNet). This network aggregates both complexity and segmentation features to produce an attention map for refining segmentation features within an Image Complexity Guided Attention (ICGA) module. We optimize the network in terms of both segmentation and image complexity prediction tasks with a combined loss function. Experimental results on the Cityscapes and CamViD datasets have shown that our ICFRNet achieves higher accuracy with a competitive efficiency for real-time segmentation.

ICFRNet: Image Complexity Prior Guided Feature Refinement for Real-time Semantic Segmentation

TL;DR

The Image Complexity prior-guided Feature Refinement Network (ICFRNet) is proposed, which aggregates both complexity and segmentation features to produce an attention map for refining segmentation features within an Image Complexity Guided Attention (ICGA) module.

Abstract

In this paper, we leverage image complexity as a prior for refining segmentation features to achieve accurate real-time semantic segmentation. The design philosophy is based on the observation that different pixel regions within an image exhibit varying levels of complexity, with higher complexities posing a greater challenge for accurate segmentation. We thus introduce image complexity as prior guidance and propose the Image Complexity prior-guided Feature Refinement Network (ICFRNet). This network aggregates both complexity and segmentation features to produce an attention map for refining segmentation features within an Image Complexity Guided Attention (ICGA) module. We optimize the network in terms of both segmentation and image complexity prediction tasks with a combined loss function. Experimental results on the Cityscapes and CamViD datasets have shown that our ICFRNet achieves higher accuracy with a competitive efficiency for real-time segmentation.
Paper Structure (13 sections, 5 equations, 3 figures, 3 tables)

This paper contains 13 sections, 5 equations, 3 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Real-time Semantic Segmentation
  4. Image Complexity
  5. Methodology
  6. Network Overview
  7. Image Complexity Guided Attention Module
  8. Dual-task Framework Optimization
  9. Experimental Results
  10. Settings
  11. Comparison With the State-of-the-Art Methods
  12. Ablation Studies
  13. Conclusion

Figures (3)

  • Figure 1: Presentation of the relationship between image complexity and semantic segmentation. The regions of high complexity marked with yellow boxes contain more details and the pixels are prone to misclassification. With the image complexity guidance, our ICFRNet pays more attention to the complex pixel regions.
  • Figure 2: Illustration of the pipeline. The input image is processed through the encoder and the obtained latent features $R$ flow into two branches ($i.e.$, IC Branch and SS Branch). D- denotes the detail sub-branch and B- denotes the boundary sub-branch. The image complexity features are aggregated into the detail and boundary sub-branches for feature refinement through our ICGA module. Also, the semantic feature from the context sub-branch is fused into detail and boundary features through the PAG xu2023pidnet and adding operation xu2023pidnet separately. IC Loss $\mathcal{L}_{c}$ and Seg Loss $\mathcal{L}_{s}$ are the loss functions of IC Branch and SS Branch.
  • Figure 3: Visualization of semantic segmentation results on the Cityscapes val set. The maps in each row from left to right are the input image, predicted pixel-wise complexity map, ground truth segmentation results, results of PIDNet, and our results. Part of the high complexity regions are marked with orange boxes in the input images and the complexity maps. The corresponding regions in the ground truth and segmentation results are marked with red boxes and zoomed in for better illustration.