Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Bidirectional Cross-Perception for Open-Vocabulary Semantic Segmentation in Remote Sensing Imagery

Jianzheng Wang, Huan Ni

TL;DR

This work tackles open-vocabulary semantic segmentation in remote sensing by addressing the limitations of training-free approaches that rely on one-way fusion and shallow post-processing. It introduces SDCI, a training-free framework with three core modules—CAF for cross-model attention fusion, BCDR for bidirectional cross-graph diffusion, and CSCP for superpixel-guided convex optimization—that jointly fuse CLIP’s semantic knowledge with DINO’s structural representations and pixel-level priors. The method demonstrates consistent performance gains across multiple datasets and prompt settings (Original vs Generalized labels), with CSCP proving especially effective at sharpening boundaries and maintaining semantic coherence. The results emphasize that integrating traditional geometric priors with deep learning representations can substantially improve boundary precision and generalization in OVSS for remote sensing, while preserving training-free flexibility.

Abstract

High-resolution remote sensing imagery is characterized by densely distributed land-cover objects and complex boundaries, which places higher demands on both geometric localization and semantic prediction. Existing training-free open-vocabulary semantic segmentation (OVSS) methods typically fuse CLIP and vision foundation models (VFMs) using "one-way injection" and "shallow post-processing" strategies, making it difficult to satisfy these requirements. To address this issue, we propose a spatial-regularization-aware dual-branch collaborative inference framework for training-free OVSS, termed SDCI. First, during feature encoding, SDCI introduces a cross-model attention fusion (CAF) module, which guides collaborative inference by injecting self-attention maps into each other. Second, we propose a bidirectional cross-graph diffusion refinement (BCDR) module that enhances the reliability of dual-branch segmentation scores through iterative random-walk diffusion. Finally, we incorporate low-level superpixel structures and develop a convex-optimization-based superpixel collaborative prediction (CSCP) mechanism to further refine object boundaries. Experiments on multiple remote sensing semantic segmentation benchmarks demonstrate that our method achieves better performance than existing approaches. Moreover, ablation studies further confirm that traditional object-based remote sensing image analysis methods leveraging superpixel structures remain effective within deep learning frameworks. Code: https://github.com/yu-ni1989/SDCI.

Bidirectional Cross-Perception for Open-Vocabulary Semantic Segmentation in Remote Sensing Imagery

TL;DR

This work tackles open-vocabulary semantic segmentation in remote sensing by addressing the limitations of training-free approaches that rely on one-way fusion and shallow post-processing. It introduces SDCI, a training-free framework with three core modules—CAF for cross-model attention fusion, BCDR for bidirectional cross-graph diffusion, and CSCP for superpixel-guided convex optimization—that jointly fuse CLIP’s semantic knowledge with DINO’s structural representations and pixel-level priors. The method demonstrates consistent performance gains across multiple datasets and prompt settings (Original vs Generalized labels), with CSCP proving especially effective at sharpening boundaries and maintaining semantic coherence. The results emphasize that integrating traditional geometric priors with deep learning representations can substantially improve boundary precision and generalization in OVSS for remote sensing, while preserving training-free flexibility.

Abstract

High-resolution remote sensing imagery is characterized by densely distributed land-cover objects and complex boundaries, which places higher demands on both geometric localization and semantic prediction. Existing training-free open-vocabulary semantic segmentation (OVSS) methods typically fuse CLIP and vision foundation models (VFMs) using "one-way injection" and "shallow post-processing" strategies, making it difficult to satisfy these requirements. To address this issue, we propose a spatial-regularization-aware dual-branch collaborative inference framework for training-free OVSS, termed SDCI. First, during feature encoding, SDCI introduces a cross-model attention fusion (CAF) module, which guides collaborative inference by injecting self-attention maps into each other. Second, we propose a bidirectional cross-graph diffusion refinement (BCDR) module that enhances the reliability of dual-branch segmentation scores through iterative random-walk diffusion. Finally, we incorporate low-level superpixel structures and develop a convex-optimization-based superpixel collaborative prediction (CSCP) mechanism to further refine object boundaries. Experiments on multiple remote sensing semantic segmentation benchmarks demonstrate that our method achieves better performance than existing approaches. Moreover, ablation studies further confirm that traditional object-based remote sensing image analysis methods leveraging superpixel structures remain effective within deep learning frameworks. Code: https://github.com/yu-ni1989/SDCI.
Paper Structure (35 sections, 19 equations, 5 figures, 3 tables)

This paper contains 35 sections, 19 equations, 5 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Methods based on improving internal mechanisms of CLIP.
  4. Methods using only VFMs.
  5. Multi-model collaborative methods.
  6. Methodology
  7. Overview
  8. CAF
  9. Semantic Branch.
  10. Structural Branch.
  11. Cross-Fusion.
  12. BCDR
  13. Semantic Graph Construction.
  14. Structural Graph Construction.
  15. Bidirectional refinement.
  16. ...and 20 more sections

Figures (5)

  • Figure 1: Observation on the boundary-enhancing effect of superpixels. (a) original remote sensing image, (b) superpixel segmentation result; (c) semantic segmentation result generated by CLIPer, (d) ground truth.
  • Figure 2: The proposed SDCI framework, in which CAF, BCDR, and CSCP serve as the core modules. CAF enables interaction between semantic and structural information by exchanging attention maps. The resulting initial logit maps are then fed into the BCDR module, which performs cross diffusion between the structural graph constructed by DINO and the semantic graph constructed by CLIP to achieve globally consistent enhancement. Finally, CSCP imposes constraints from superpixel structures and, through a convex optimization process, fuses the predictions from the two branches to generate the final segmentation map.
  • Figure 3: Visualization observations of CAF bidirectional cross-fusion. In (a), by introducing CLIP semantic priors, DINO's originally scattered similarity-based structures are semantically aggregated, endowing the unsupervised features with explicit category guidance; in (b), after injecting DINO structural information, targets similar to the current semantic category are enhanced, enabling the spatial propagation of CLIP semantic cues.
  • Figure 4: Visual comparison with existing methods.
  • Figure 5: The visualization results of the ablation study. This visualization shows that, as the proposed modules are progressively added, the segmentation performance is continuously improved. (a) original image, (b) results of the baseline model, (c) results of baseline + CAF, (d) results of baseline + CAF + BCDR, (e) full model with CSCP (SDCI-v1), (f) full model with CSCP (SDCI-v2), (g) ground truth.