WaveRNet: Wavelet-Guided Frequency Learning for Multi-Source Domain-Generalized Retinal Vessel Segmentation

TL;DR

WaveRNet introduces a frequency-aware framework for multi-source domain-generalized retinal vessel segmentation. It combines a Spectral-guided Domain Modulator to separate illumination-stable low-frequency content from contrast-sensitive high-frequency edges, a Frequency-Adaptive Domain Fusion for test-time domain weighting, and a Hierarchical Mask-Prompt Refiner to progressively recover fine vessel structures beyond SAM's upsampling. Through Leave-One-Domain-Out evaluations on four public datasets, WaveRNet achieves state-of-the-art cross-domain generalization, with ablations confirming the complementary contributions of SDM, FADF, and HMPR. The approach promises robust deployment of automated vascular analysis across heterogeneous ophthalmic imaging environments.

Abstract

Domain-generalized retinal vessel segmentation is critical for automated ophthalmic diagnosis, yet faces significant challenges from domain shift induced by non-uniform illumination and varying contrast, compounded by the difficulty of preserving fine vessel structures. While the Segment Anything Model (SAM) exhibits remarkable zero-shot capabilities, existing SAM-based methods rely on simple adapter fine-tuning while overlooking frequency-domain information that encodes domain-invariant features, resulting in degraded generalization under illumination and contrast variations. Furthermore, SAM's direct upsampling inevitably loses fine vessel details. To address these limitations, we propose WaveRNet, a wavelet-guided frequency learning framework for robust multi-source domain-generalized retinal vessel segmentation. Specifically, we devise a Spectral-guided Domain Modulator (SDM) that integrates wavelet decomposition with learnable domain tokens, enabling the separation of illumination-robust low-frequency structures from high-frequency vessel boundaries while facilitating domain-specific feature generation. Furthermore, we introduce a Frequency-Adaptive Domain Fusion (FADF) module that performs intelligent test-time domain selection through wavelet-based frequency similarity and soft-weighted fusion. Finally, we present a Hierarchical Mask-Prompt Refiner (HMPR) that overcomes SAM's upsampling limitation through coarse-to-fine refinement with long-range dependency modeling. Extensive experiments under the Leave-One-Domain-Out protocol on four public retinal datasets demonstrate that WaveRNet achieves state-of-the-art generalization performance. The source code is available at https://github.com/Chanchan-Wang/WaveRNet.

Figures (3)

• Figure 1: Overview of the proposed WaveRNet framework. The Spectral-guided Domain Modulator (SDM) decomposes features into high-frequency and low-frequency components through learnable wavelet transform, followed by domain-specific modulation using learnable domain tokens. During inference, Frequency-Adaptive Domain Fusion (FADF) computes wavelet-based frequency similarity for intelligent test-time domain selection. The Hierarchical Mask-Prompt Refiner (HMPR) progressively refines segmentation through coarse-to-fine mask generation with cross-attention and hierarchical upsampling.
• Figure 2: Qualitative comparison of segmentation results under single-domain training. From left to right: input image, ground truth, UNeXt, Swin-UNet, ResUNet++, SAM-FT, MedSAM-FT, and WaveRNet (Ours). Each row represents a different dataset: DRIVE ($\mathcal{S}_1$), STARE ($\mathcal{S}_2$), CHASE_DB1 ($\mathcal{S}_3$), and RECOVERY-FA19 ($\mathcal{S}_4$). White overlays indicate the predicted vessel segmentation masks.
• Figure 3: Qualitative comparison of segmentation results on unseen target domains under the LODO protocol. From left to right: input image, ground truth, UNet++, Swin-UNet, ResUNet++, SAM-FT, MedSAM-FT, and WaveRNet (Ours). Each row represents a different unseen target domain: DRIVE ($\mathcal{T}=\mathcal{S}_1$), CHASE_DB1 ($\mathcal{T}=\mathcal{S}_3$), STARE ($\mathcal{T}=\mathcal{S}_2$), and RECOVERY-FA19 ($\mathcal{T}=\mathcal{S}_4$). White overlays indicate the predicted vessel segmentation masks.