SegAssess: Panoramic quality mapping for robust and transferable unsupervised segmentation assessment

TL;DR

SegAssess reframes segmentation quality assessment as Panoramic Quality Mapping (PQM), enabling pixel-level classification into TP, FP, TN, and FN to produce a complete quality map without ground truth. Built on an enhanced HQ-SAM backbone, it introduces an Edge Guided Compaction (EGC) with Aggregated Semantic Filter (ASF) and an Augmented Mixup Sampling (AMS) strategy to boost edge accuracy and cross-domain generalization. The approach achieves state-of-the-art performance on multiple remote sensing SQA benchmarks and demonstrates strong zero-shot transfer to unseen masks from both model- and human-sourced origins, addressing transferability gaps in prior methods. Collectively, PQM and SegAssess offer a robust, transferable framework for unsupervised SQA in large-scale geospatial analysis, with practical implications for quality control across diverse RS datasets and sensor modalities.

Abstract

High-quality image segmentation is fundamental to pixel-level geospatial analysis in remote sensing, necessitating robust segmentation quality assessment (SQA), particularly in unsupervised settings lacking ground truth. Although recent deep learning (DL) based unsupervised SQA methods show potential, they often suffer from coarse evaluation granularity, incomplete assessments, and poor transferability. To overcome these limitations, this paper introduces Panoramic Quality Mapping (PQM) as a new paradigm for comprehensive, pixel-wise SQA, and presents SegAssess, a novel deep learning framework realizing this approach. SegAssess distinctively formulates SQA as a fine-grained, four-class panoramic segmentation task, classifying pixels within a segmentation mask under evaluation into true positive (TP), false positive (FP), true negative (TN), and false negative (FN) categories, thereby generating a complete quality map. Leveraging an enhanced Segment Anything Model (SAM) architecture, SegAssess uniquely employs the input mask as a prompt for effective feature integration via cross-attention. Key innovations include an Edge Guided Compaction (EGC) branch with an Aggregated Semantic Filter (ASF) module to refine predictions near challenging object edges, and an Augmented Mixup Sampling (AMS) training strategy integrating multi-source masks to significantly boost cross-domain robustness and zero-shot transferability. Comprehensive experiments demonstrate that SegAssess achieves state-of-the-art (SOTA) performance and exhibits remarkable zero-shot transferability to unseen masks. The code is available at https://github.com/Yangbn97/SegAssess.

Figures (15)

• Figure 1: Schematic diagrams of different segmentation quality assessment (SQA) paradigms (second row) and visual comparison between proposed Panoramic Quality Mapping scheme via SegAssess model (fourth row) and other deep learning based unsupervised SQA methods (third row). ASF: Aggregated Semantic Filter.
• Figure 2: Visual comparison between SegAssess (PQM approach) and prior work (AQSNet). SegAssess and AQSNet were firstly trained with model generated masks (first row), and then zero-shot applied to unseen masks from historial human annotations (second row). (a) Input image with mask under evaluation (Orange contour) and ground truth (magenta contour). (b) Ground truth PQM assessment map (4-class). (c) AQSNet result (FP/FN errors only). (d) SegAssess PQM result (4-class). Colors represent TP, FP, TN and FN. Pink overlapping FP/FN predictions by AQSNet, highlighting spatial ambiguity not present in the SegAssess PQM output. Golden highlights AQSNet's failure in zero-shot transferring from model-source masks with edge-clustered discrepancies between unchecked and GT masks to unseen human-source masks with instance-wise discrepancies.
• Figure 3: Overview of the SegAssess framework architecture for Panoramic Quality Mapping. SegAssess employs a promptable HQ-SAM backbone for coarse PQM assessment, followed by an Edge Guided Compaction (EGC) branch (orange box) incorporating an Aggregated Semantic Filter (ASF) module (pink rectangle) for result refinement.
• Figure 4: Detailed architecture of the Aggregated Semantic Filter (ASF) module. It integrates three perception mechanisms: spectral-spatial attention, a spatial decomposed filter, and a multiple perceive fields filter using atrous convolutions.
• Figure 5: Supporting strategies for SegAssess training. (a) The Augmented Mixup Sampling (AMS) strategy used during training to enhance robustness by generating diverse masks online using multiple frozen segmentation models. (b) Overview of the multi-component loss function calculation used for supervision.