Towards Any-Quality Image Segmentation via Generative and Adaptive Latent Space Enhancement

TL;DR

GleSAM++ tackles the challenge of segmenting images across arbitrary quality by integrating a pretrained latent diffusion model into the SAM framework. It introduces Latent Space Alignment with Feature Distribution Alignment and Channel Replication and Expansion to bridge the gap between diffusion latents and SAM features, and a Degradation-aware Adaptive Enhancement module that explicitly predicts degradation level and adaptively controls denoising strength. The approach is trained with a two-step LoRA-based fine-tuning strategy and evaluated on the newly constructed LQSeg dataset, showing strong robustness to seen and unseen degradations while preserving performance on clear images. The work demonstrates practical gains in robustness for real-world scenarios and offers a scalable path toward degradation-aware segmentation in multimodal and downstream tasks.

Abstract

Segment Anything Models (SAMs), known for their exceptional zero-shot segmentation performance, have garnered significant attention in the research community. Nevertheless, their performance drops significantly on severely degraded, low-quality images, limiting their effectiveness in real-world scenarios. To address this, we propose GleSAM++, which utilizes Generative Latent space Enhancement to boost robustness on low-quality images, thus enabling generalization across various image qualities. Additionally, to improve compatibility between the pre-trained diffusion model and the segmentation framework, we introduce two techniques, i.e., Feature Distribution Alignment (FDA) and Channel Replication and Expansion (CRE). However, the above components lack explicit guidance regarding the degree of degradation. The model is forced to implicitly fit a complex noise distribution that spans conditions from mild noise to severe artifacts, which substantially increases the learning burden and leads to suboptimal reconstructions. To address this issue, we further introduce a Degradation-aware Adaptive Enhancement (DAE) mechanism. The key principle of DAE is to decouple the reconstruction process for arbitrary-quality features into two stages: degradation-level prediction and degradation-aware reconstruction. Our method can be applied to pre-trained SAM and SAM2 with only minimal additional learnable parameters, allowing for efficient optimization. Extensive experiments demonstrate that GleSAM++ significantly improves segmentation robustness on complex degradations while maintaining generalization to clear images. Furthermore, GleSAM++ also performs well on unseen degradations, underscoring the versatility of our approach and dataset.

Figures (13)

• Figure 1: Qualitative results on low-quality images with varying degradation levels from an unseen dataset. To generate images with different degradation levels, we progressively added Gaussian Noise, Re-sampling Noise, and more severe Gaussian noise to an image. Results indicate that the baseline SAM sam shows limited robustness to degradation. Although RobustSAM robustsam retains some resilience against simpler degradations, it struggles with more complex and unfamiliar degradations. In contrast, our method consistently demonstrates strong robustness across images of varying quality.
• Figure 2: The visualization of latent features: (a) low-quality (LQ) images, (b) the SAM's latent features extracted from LQ images, which contain excessive noise and compromise the original representations, (c) enhanced representation by our GleSAM++, exhibiting more salient and well-preserved semantics and (d) the high-quality (HQ) features of the corresponding clear images, which are more salient than LQ ones.
• Figure 3: Given an input image, GleSAM++ performs accurate segmentation through image encoding, generative and adaptive latent space enhancement, and mask decoding. During training, with HQ-LQ image pairs as input, we adaptively reconstruct high-quality representations in the SAM's latent space by efficiently fine-tuning a generative denoising U-Net with LoRA layers. Degradation-aware adaptive enhancement is used to explicitly estimate the degradation level of the input features and uses this information to dynamically regulate the denoising strength. Latent space alignment is used to bridge the feature distribution and structural gaps between the pre-trained latent diffusion model and SAM. Subsequently, the decoder is fine-tuned with segmentation loss to align the enhanced latent representations. Built upon SAMs, GleSAM++ inherits prompt-based segmentation and performs well on images of any quality.
• Figure 4: Aligning feature distributions before (a) and after (b) FDA.
• Figure 5: Details of the degradation-aware prediction module.