SegEarth-OV3: Exploring SAM 3 for Open-Vocabulary Semantic Segmentation in Remote Sensing Images

TL;DR

Open-Vocabulary Semantic Segmentation (OVSS) in remote sensing is challenged by coarse localization and pipeline complexity. The authors repurpose SAM 3 into a training-free SegEarth-OV3 framework, combining dual-head mask fusion and presence-guided filtering. Their experiments across 17 remote sensing datasets (and general scene benchmarks) show state-of-the-art zero-shot performance, with SegEarth-OV3 sometimes beating Oracle and certain supervised baselines. The work demonstrates the practical viability of SAM-based, training-free OVSS in geospatial analysis and provides code for reproducibility.

Abstract

Most existing methods for training-free Open-Vocabulary Semantic Segmentation (OVSS) are based on CLIP. While these approaches have made progress, they often face challenges in precise localization or require complex pipelines to combine separate modules, especially in remote sensing scenarios where numerous dense and small targets are present. Recently, Segment Anything Model 3 (SAM 3) was proposed, unifying segmentation and recognition in a promptable framework. In this paper, we present a preliminary exploration of applying SAM 3 to the remote sensing OVSS task without any training. First, we implement a mask fusion strategy that combines the outputs from SAM 3's semantic segmentation head and the Transformer decoder (instance head). This allows us to leverage the strengths of both heads for better land coverage. Second, we utilize the presence score from the presence head to filter out categories that do not exist in the scene, reducing false positives caused by the vast vocabulary sizes and patch-level processing in geospatial scenes. We evaluate our method on extensive remote sensing datasets. Experiments show that this simple adaptation achieves promising performance, demonstrating the potential of SAM 3 for remote sensing OVSS. Our code is released at https://github.com/earth-insights/SegEarth-OV-3.

Figures (4)

• Figure 1: The Transformer decoder of SAM 3 excels at delineating countable objects but produces fragmented masks for amorphous regions, while the semantic head preserves continuity for amorphous regions but lacks boundary precision for small targets. (Left: Remote sensing image. Middle: Prediction of Transformer Decoder. Right: Prediction of semantic segmentation head.)
• Figure 2: The overall inference pipeline of SegEarth-OV3. Given an input image and a list of text prompts, we leverage SAM 3's decoupled outputs. The pipeline involves: (1) instance aggregation to consolidate sparse object queries; (2) dual-head mask fusion to combine the fine-grained instance details with the global coverage of the semantic head; and (3) presence-guided filtering (using the presence score) to suppress false positives from absent categories. denotes the element-wise maximum operation, and denotes multiplication.
• Figure 3: Impact of vocabulary size and our filtering strategy. Querying a vast vocabulary introduces severe noise due to distracting candidates (b to c). Our presence-guided filtering leverages presence scores to suppress absent categories, effectively eliminating interference and restoring segmentation quality.
• Figure 4: Inference results of SegEarth-OV3 on a remote sensing image exceeding 10k$\times$10k resolution. The image originates from chen2023land.