DeepMerge: Deep-Learning-Based Region-Merging for Image Segmentation

TL;DR

DeepMerge tackles the challenge of segmenting large-area VHR remote sensing imagery with interpretable segmentation scales. It fuses a deep transformer-based similarity learner (S2Former) with region adjacency graphs, using shift-scale inputs generated by BTS to learn cross-scale super-pixel similarities and perform region merging with a fixed threshold of $0.5$. The key contributions are the novel shift-scale attention, scale-wise pooling, auxiliary and segment-based features, and a loss design that yields an interpretable optimal scale around $0.5$, achieving state-of-the-art F-values (e.g., $0.9550$) and low total error ($0.0895$) on a $0.55$ m Phoenix dataset. The approach reduces the need for manual scale tuning, enhances interpretability, and demonstrates strong transferability across urban and rural scenes, with potential for unsupervised extensions in future work.

Abstract

Image segmentation aims to partition an image according to the objects in the scene and is a fundamental step in analysing very high spatial-resolution (VHR) remote sensing imagery. Current methods struggle to effectively consider land objects with diverse shapes and sizes. Additionally, the determination of segmentation scale parameters frequently adheres to a static and empirical doctrine, posing limitations on the segmentation of large-scale remote sensing images and yielding algorithms with limited interpretability. To address the above challenges, we propose a deep-learning-based region merging method dubbed DeepMerge to handle the segmentation of complete objects in large VHR images by integrating deep learning and region adjacency graph (RAG). This is the first method to use deep learning to learn the similarity and merge similar adjacent super-pixels in RAG. We propose a modified binary tree sampling method to generate shift-scale data, serving as inputs for transformer-based deep learning networks, a shift-scale attention with 3-Dimension relative position embedding to learn features across scales, and an embedding to fuse learned features with hand-crafted features. DeepMerge can achieve high segmentation accuracy in a supervised manner from large-scale remotely sensed images and provides an interpretable optimal scale parameter, which is validated using a remote sensing image of 0.55 m resolution covering an area of 5,660 km^2. The experimental results show that DeepMerge achieves the highest F value (0.9550) and the lowest total error TE (0.0895), correctly segmenting objects of different sizes and outperforming all competing segmentation methods.

Figures (20)

• Figure 1: Comparison of different region-merging based segmentation methods in region-merging. (a) Original remote sensing image. (b) Initial over-segmented super-pixels. (c) Region-merging results of SOTA method BCMS zhang2013boundary. (d) Region-merging results of SOTA method Local-SA yang2017region. (e) Region-merging results of the proposed method DeepMerge.
• Figure 2: The shift-scale presentations in the DeepMerge.
• Figure 3: The construction of RAG and a region-merging step in the RAG.
• Figure 4: The workflow of DeepMerge for super-pixel segmentation in high-resolution remote sensing imagery. W means shared weights. Aux is the loss function auxiliary module. FC is the fully connected layers. SFE is the segment-absed feature embedding module.
• Figure 5: Sample collection graphical interface and sample pairs. (a) is the graphical interface of sample collection. (b) and (c) are the positive sample pairs and negative sample pairs, respectively.