Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Bridging Structure and Appearance: Topological Features for Robust Self-Supervised Segmentation

Haotang Li, Zhenyu Qi, Hao Qin, Huanrui Yang, Sen He, Kebin Peng

TL;DR

Self-supervised semantic segmentation often falters under appearance ambiguities due to reliance on unstable cues. GASeg addresses this by bridging geometry and appearance through topology, via the Differentiable Box-Counting module (DBC), Topological Augmentation (TopoAug), and the multi-objective GALoss that aligns cross-modal representations. The approach yields state-of-the-art results across four benchmarks (COCO-Stuff, Cityscapes, Potsdam, PASCAL VOC) and multiple backbones, validating the effectiveness of multi-scale topological statistics and depth-based geometry. This work highlights the practical impact of topological invariants for robust visual understanding in dense segmentation tasks.

Abstract

Self-supervised semantic segmentation methods often fail when faced with appearance ambiguities. We argue that this is due to an over-reliance on unstable, appearance-based features such as shadows, glare, and local textures. We propose \textbf{GASeg}, a novel framework that bridges appearance and geometry by leveraging stable topological information. The core of our method is Differentiable Box-Counting (\textbf{DBC}) module, which quantifies multi-scale topological statistics from two parallel streams: geometric-based features and appearance-based features. To force the model to learn these stable structural representations, we introduce Topological Augmentation (\textbf{TopoAug}), an adversarial strategy that simulates real-world ambiguities by applying morphological operators to the input images. A multi-objective loss, \textbf{GALoss}, then explicitly enforces cross-modal alignment between geometric-based and appearance-based features. Extensive experiments demonstrate that GASeg achieves state-of-the-art performance on four benchmarks, including COCO-Stuff, Cityscapes, and PASCAL, validating our approach of bridging geometry and appearance via topological information.

Bridging Structure and Appearance: Topological Features for Robust Self-Supervised Segmentation

TL;DR

Self-supervised semantic segmentation often falters under appearance ambiguities due to reliance on unstable cues. GASeg addresses this by bridging geometry and appearance through topology, via the Differentiable Box-Counting module (DBC), Topological Augmentation (TopoAug), and the multi-objective GALoss that aligns cross-modal representations. The approach yields state-of-the-art results across four benchmarks (COCO-Stuff, Cityscapes, Potsdam, PASCAL VOC) and multiple backbones, validating the effectiveness of multi-scale topological statistics and depth-based geometry. This work highlights the practical impact of topological invariants for robust visual understanding in dense segmentation tasks.

Abstract

Self-supervised semantic segmentation methods often fail when faced with appearance ambiguities. We argue that this is due to an over-reliance on unstable, appearance-based features such as shadows, glare, and local textures. We propose \textbf{GASeg}, a novel framework that bridges appearance and geometry by leveraging stable topological information. The core of our method is Differentiable Box-Counting (\textbf{DBC}) module, which quantifies multi-scale topological statistics from two parallel streams: geometric-based features and appearance-based features. To force the model to learn these stable structural representations, we introduce Topological Augmentation (\textbf{TopoAug}), an adversarial strategy that simulates real-world ambiguities by applying morphological operators to the input images. A multi-objective loss, \textbf{GALoss}, then explicitly enforces cross-modal alignment between geometric-based and appearance-based features. Extensive experiments demonstrate that GASeg achieves state-of-the-art performance on four benchmarks, including COCO-Stuff, Cityscapes, and PASCAL, validating our approach of bridging geometry and appearance via topological information.
Paper Structure (19 sections, 11 equations, 4 figures, 4 tables)

This paper contains 19 sections, 11 equations, 4 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. GASeg
  4. Overall Framework
  5. Differentiable Box-Counting Module
  6. Appearance Domain
  7. Geometric Domain
  8. Projection
  9. Topological Augmentation
  10. Loss Function
  11. Evaluation
  12. Experiment Setup
  13. Implementation Details
  14. Datasets and Metrics
  15. Quantitative Analysis
  16. ...and 4 more sections

Figures (4)

  • Figure 1: Qualitative comparison of semantic segmentation results on an image from the COCO-Stuff dataset. Our method (GASeg) successfully disambiguates the tree trunk and leaves from the background, overcoming appearance ambiguities (e.g., shadows, textures) that cause segmentation failures in state-of-the-art methods like STEGO and EAGLE.
  • Figure 2: The overall architecture of the GASeg framework. The diagram illustrates both the training and inference pipelines, highlighting the three core components: (a) the Topological Augmentation module, (b) the Differentiable Box-Counting module for learning and bridging geometric and appearance priors, and (c) the multi-objective GALoss function.
  • Figure 3: Qualitative semantic segmentation comparisons on the (a) COCO-Stuff-27 COCOStuff (a - h) and (b) Cityscapes Cityscapes (i - k) datasets. GASeg consistently produces more spatially coherent and accurate masks compared to STEGO and EAGLE.
  • Figure 4: Efficiency analysis plotting model accuracy mIoU on COCO-Stuff against computational cost. The size of each bubble corresponds to the model's parameter count. Our models (Ours, ViT-S/8, and ViT-B/8) establish a new SOTA efficiency frontier, achieving higher accuracy for their respective computational brackets.