Steerable Pyramid Weighted Loss: Multi-Scale Adaptive Weighting for Semantic Segmentation

TL;DR

Semantic segmentation often struggles with fine structures and precise boundaries under standard losses. The authors introduce SPW loss, a steerable pyramid–based, adaptive weighting scheme that generates multi-scale weight maps from both ground truth and model predictions to refine learning where it matters. Key contributions include multi-scale steerable pyramid weighting, analytic-signal–based envelope weighting, online weight map computation without gradient tracking, and comprehensive evaluation across SNEMI3D, GlaS, and DRIVE showing superior performance against 11 baselines with modest overhead. The method advances multi-scale feature representation in loss design and offers practical benefits for challenging segmentation tasks, with future work aimed at 3D extensions and broader domain generalization.

Abstract

Semantic segmentation is a core task in computer vision with applications in biomedical imaging, remote sensing, and autonomous driving. While standard loss functions such as cross-entropy and Dice loss perform well in general cases, they often struggle with fine structures, particularly in tasks involving thin structures or closely packed objects. Various weight map-based loss functions have been proposed to address this issue by assigning higher loss weights to pixels prone to misclassification. However, these methods typically rely on precomputed or runtime-generated weight maps based on distance transforms, which impose significant computational costs and fail to adapt to evolving network predictions. In this paper, we propose a novel steerable pyramid-based weighted (SPW) loss function that efficiently generates adaptive weight maps. Unlike traditional boundary-aware losses that depend on static or iteratively updated distance maps, our method leverages steerable pyramids to dynamically emphasize regions across multiple frequency bands (capturing features at different scales) while maintaining computational efficiency. Additionally, by incorporating network predictions into the weight computation, our approach enables adaptive refinement during training. We evaluate our method on the SNEMI3D, GlaS, and DRIVE datasets, benchmarking it against 11 state-of-the-art loss functions. Our results demonstrate that the proposed SPW loss function achieves superior pixel precision and segmentation accuracy with minimal computational overhead. This work provides an effective and efficient solution for improving semantic segmentation, particularly for applications requiring multiscale feature representation. The code is avaiable at https://anonymous.4open.science/r/SPW-0884

Figures (4)

• Figure 1: Illustration of the Steerable Pyramid-based Weighted (SPW) Loss. The weighted maps ($\mathbf{Y}_{SPW}$, $\mathbf{P}_{SPW}$) derived from the ground truth label ($\mathbf{Y}$) and the predicted probability map ($\mathbf{P}$) are incorporated into a weighted cross-entropy loss. To compute $\mathbf{Y}_{SPW}$, $\mathbf{Y}$ is first decomposed using a steerable pyramid, producing subbands at multiple scales and orientations ($\mathbf{Y}_{B_{ik}}$). The amplitudes of the analytic signal from all subbands are then upsampled and combined to form the final SPW map.
• Figure 2: Qualitative results of different loss functions on the SNEMI3D dataset. Red: false positive regions; Blue: false negative regions. Green box: challenging segmentation errors that are successfully addressed by SPW loss.
• Figure 3: Qualitative results of different loss functions on the GlaS dataset. Red: false positive regions; Blue: false negative regions. Green box: challenging segmentation errors that are successfully addressed by SPW loss.
• Figure 4: Qualitative results of different loss functions on the DRIVE dataset. Red: false positive regions; Blue: false negative regions. Green box: challenging segmentation errors that are successfully addressed by SPW loss.