Taxonomy-Aware Continual Semantic Segmentation in Hyperbolic Spaces for Open-World Perception

TL;DR

This work proposes Taxonomy-Oriented Poincaré-regularized Incremental-Class Segmentation (TOPICS) that learns feature embeddings in hyperbolic space following explicit taxonomy-tree structures and maintains implicit class relational constraints on the geometric basis of the Poincaré ball.

Abstract

Semantic segmentation models are typically trained on a fixed set of classes, limiting their applicability in open-world scenarios. Class-incremental semantic segmentation aims to update models with emerging new classes while preventing catastrophic forgetting of previously learned ones. However, existing methods impose strict rigidity on old classes, reducing their effectiveness in learning new incremental classes. In this work, we propose Taxonomy-Oriented Poincaré-regularized Incremental-Class Segmentation (TOPICS) that learns feature embeddings in hyperbolic space following explicit taxonomy-tree structures. This supervision provides plasticity for old classes, updating ancestors based on new classes while integrating new classes at fitting positions. Additionally, we maintain implicit class relational constraints on the geometric basis of the Poincaré ball. This ensures that the latent space can continuously adapt to new constraints while maintaining a robust structure to combat catastrophic forgetting. We also establish eight realistic incremental learning protocols for autonomous driving scenarios, where novel classes can originate from known classes or the background. Extensive evaluations of TOPICS on the Cityscapes and Mapillary Vistas 2.0 benchmarks demonstrate that it achieves state-of-the-art performance. We make the code and trained models publicly available at http://topics.cs.uni-freiburg.de.

Figures (13)

• Figure 1: TOPICS leverages the explicit class taxonomy (black) and implicit relations (red and green) in hyperbolic space to balance rigidity and plasticity in taxonomic class-incremental semantic segmentation.
• Figure 2: During base training of TOPICS, features are mapped onto the Poincaré ball before the class hierarchy is explicitly enforced with $\mathcal{L}_{hier}$. In incremental steps, the old model is used to generate pseudo-labels of old classes (PL) and to regularize the last layer's weights with $\mathcal{L}_{rel}$ and feature radii with $\mathcal{L}_{dist}$.
• Figure 3: Visualization of the class taxonomic tree $\mathcal{H}$: a) Novel classes originate from the background and b) Novel classes originate from known classes. Base classes ($\mathcal{C}^{1}$) are colored in blue whereas novel classes ($\mathcal{C}^{2:T}$) are colored in orange. Novel ancestor nodes are visualized with orange outlines.
• Figure 4: Visualization of a hyperplane on an upper sheet of a two-sheeted hyperboloid which is projected on a 2D Poincaré ball. The hyperplane has an offset of $o_i$ from the center and an orientation $r_i$.
• Figure 5: Performance at every increment on Cityscapes 10-1 (10 task) setting. TOPICS$_{b2}$ is trained for half iterations in base training to illustrate knowledge retention with lower base training performance.