WaveSeg: Enhancing Segmentation Precision via High-Frequency Prior and Mamba-Driven Spectrum Decomposition

TL;DR

WaveSeg tackles the decoder bottleneck in semantic segmentation by introducing a dual-domain decoder that uses a learnable high-frequency prior from the input image and a Spectrum Decomposition Attention block based on the Mamba selective state-space model. The approach fuses spatial and wavelet-domain refinements through a Wavelet Mixer and a Local Perception Block, with reparameterized convolutions preserving low-frequency semantics. Empirical results across ADE20K, Cityscapes, COCO-Stuff 164K, and PASCAL VOC2012 demonstrate superior accuracy and efficiency, often outperforming state-of-the-art methods at lower computational cost. The work offers a practical, annotation-free mechanism to recover fine details and boundaries in dense predictions, with potential for further optimization in the scan order of the Mamba module.

Abstract

While recent semantic segmentation networks heavily rely on powerful pretrained encoders, most employ simplistic decoders, leading to suboptimal trade-offs between semantic context and fine-grained detail preservation. To address this, we propose a novel decoder architecture, WaveSeg, which jointly optimizes feature refinement in spatial and wavelet domains. Specifically, high-frequency components are first learned from input images as explicit priors to reinforce boundary details at early stages. A multi-scale fusion mechanism, Dual Domain Operation (DDO), is then applied, and the novel Spectrum Decomposition Attention (SDA) block is proposed, which is developed to leverage Mamba's linear-complexity long-range modeling to enhance high-frequency structural details. Meanwhile, reparameterized convolutions are applied to preserve low-frequency semantic integrity in the wavelet domain. Finally, a residual-guided fusion integrates multi-scale features with boundary-aware representations at native resolution, producing semantically and structurally rich feature maps. Extensive experiments on standard benchmarks demonstrate that WaveSeg, leveraging wavelet-domain frequency prior with Mamba-based attention, consistently outperforms state-of-the-art approaches both quantitatively and qualitatively, achieving efficient and precise segmentation.

Figures (10)

• Figure 1: Comparisons of the proposed high-frequency prior guidance strategy with ground-truth-based guidance and conventional vanilla segmentation architectures.
• Figure 2: Overall segmentation pipeline. The input image is first encoded into multi-level features by a pretrained backbone. These hierarchical features are then progressively enhanced and fused in our proposed dual-domain decoder.
• Figure 3: Overall architecture of our proposed dual-domain decoder. Feature maps $F_2$, $F_3$, and $F_4$ are extracted from different stages of the pretrained backbone. The decoder incorporates high-frequency prior guidance and multi-scale feature enhancement to progressively refine the semantic representations and produce accurate segmentation results.
• Figure 4: The structure of our proposed Spectrum Decomposition Attention (SDA) block. The Wavelet Mixer (WM) operates in the wavelet domain to model high-frequency structural details, while the Local Perception Block (LPB) works in the spatial domain to enhance local contextual understanding. The two branches are fused to effectively combine spatial and frequency cues for improved feature representation.
• Figure 5: Feature map visualizations before and after applying our high-frequency prior modulation (FM). The enhanced features guided by $x^H$ exhibit clearer object boundaries and richer structural details across different levels, demonstrating the effectiveness of our high-frequency prior.