PolSAM: Polarimetric Scattering Mechanism Informed Segment Anything Model

TL;DR

PolSAM addresses the challenge of PolSAR segmentation by integrating domain physics with a large pre-trained segmentation model. It introduces Microwave Vision Data (MVD), a compact, interpretable scattering representation, and two fusion prompts—FFP and SFP—to merge MVD with Pauli-based imagery while keeping the SAM encoder frozen via adapters. On PhySAR-Seg, PolSAM achieves state-of-the-art performance over SAM-based and multimodal baselines, while reducing data storage and speeding inference. This work delivers a scalable, interpretable, and efficient framework for PolSAR segmentation and provides accessible code and datasets for further research.

Abstract

PolSAR data presents unique challenges due to its rich and complex characteristics. Existing data representations, such as complex-valued data, polarimetric features, and amplitude images, are widely used. However, these formats often face issues related to usability, interpretability, and data integrity. Most feature extraction networks for PolSAR are small, limiting their ability to capture features effectively. To address these issues, We propose the Polarimetric Scattering Mechanism-Informed SAM (PolSAM), an enhanced Segment Anything Model (SAM) that integrates domain-specific scattering characteristics and a novel prompt generation strategy. PolSAM introduces Microwave Vision Data (MVD), a lightweight and interpretable data representation derived from polarimetric decomposition and semantic correlations. We propose two key components: the Feature-Level Fusion Prompt (FFP), which fuses visual tokens from pseudo-colored SAR images and MVD to address modality incompatibility in the frozen SAM encoder, and the Semantic-Level Fusion Prompt (SFP), which refines sparse and dense segmentation prompts using semantic information. Experimental results on the PhySAR-Seg datasets demonstrate that PolSAM significantly outperforms existing SAM-based and multimodal fusion models, improving segmentation accuracy, reducing data storage, and accelerating inference time. The source code and datasets will be made publicly available at https://github.com/XAI4SAR/PolSAM.

Figures (13)

• Figure 1: (a) Different input representations of PolSAR data and their corresponding attributes, including lightweight (Lw), physical interpretability (Phys. i.), and information integrity (Inf. int.), are explored. (b) Scattering mechanisms depict polarization characteristics with physical interpretability and exhibit a strong correlation with semantic information. (c) Original SAM architecture; (d) Our PolSAM model.
• Figure 2: Data processing pipeline: unsupervised GD-Wishart classification, MVD generation through re-clustering of scattering mechanisms, and semantic annotation, illustrated with the PhySAR-Seg-1 dataset.
• Figure 3: Semantic label clustering of data PhySAR-Seg-1.
• Figure 4: Visualization of the PhySAR-Seg-1 dataset, including the pseudo-colored image, MVD, semantic label, MVD legend, and semantic legend. The white dashed lines on the pseudo-colored image divide the dataset into train, val, and test sets.
• Figure 5: Visualization of the PhySAR-Seg-2 dataset, including the pseudo-colored image, MVD, semantic label, MVD legend, and semantic legend. The white dashed lines on the pseudo-colored image divide the dataset into train and val sets.