Learning Ordinality in Semantic Segmentation

TL;DR

This work addresses the limited use of inter-class ordinality in semantic segmentation by formalizing two consistency notions for structured observations: representation consistency (pixel-level unimodal, monotone predictions) and structural consistency (neighborhood-level ordinal coherence). It introduces two regularizers, $L_{\text{CSNP}}$ and $L_{\text{CSDT}}$, plus an adapted $L_{\text{O2}}$ framework, and a distance-transform-based mechanism, with extensions to partially ordered domains. Evaluations on five biomedical datasets and two autonomous-driving datasets show improved ordinal metrics and Dice scores, including up to $15.7\%$ relative Dice improvement in some out-of-distribution settings, while incurring no inference-time cost. The results demonstrate that encoding spatial ordinal relationships enhances generalization and supports more structured image representations for segmentation.

Abstract

Semantic segmentation consists of predicting a semantic label for each image pixel. While existing deep learning approaches achieve high accuracy, they often overlook the ordinal relationships between classes, which can provide critical domain knowledge (e.g., the pupil lies within the iris, and lane markings are part of the road). This paper introduces novel methods for spatial ordinal segmentation that explicitly incorporate these inter-class dependencies. By treating each pixel as part of a structured image space rather than as an independent observation, we propose two regularization terms and a new metric to enforce ordinal consistency between neighboring pixels. Two loss regularization terms and one metric are proposed for structural ordinal segmentation, which penalizes predictions of non-ordinal adjacent classes. Five biomedical datasets and multiple configurations of autonomous driving datasets demonstrate the efficacy of the proposed methods. Our approach achieves improvements in ordinal metrics and enhances generalization, with up to a 15.7% relative increase in the Dice coefficient. Importantly, these benefits come without additional inference time costs. This work highlights the significance of spatial ordinal relationships in semantic segmentation and provides a foundation for further exploration in structured image representations.

Figures (12)

• Figure 1: Example of (a) segmentation masks and (b) hypothetical non-constrained model outputs for an ordinal problem with three distinct classes, $\{\mathcal{C}_1,\mathcal{C}_2,\mathcal{C}_3\}$, where such an order is defined that $\mathcal{C}_1 \supset \mathcal{C}_2 \supset \mathcal{C}_3$, therefore, an area segmented as $\mathcal{C}_1$ can only possibly have a direct boundary with areas segmented as $\mathcal{C}_2$, whereas $\mathcal{C}_2$ can have boundaries both with $\mathcal{C}_1$ and $\mathcal{C}_3$.
• Figure 2: Example of possible (a) multimodal and (b) unimodal output probability distributions for a given pixel.
• Figure 3: Illustration of possible ordinal (in)consistencies. (a) Ordinal representation consistency. (b) Ordinal structure inconsistency. (c) Ordinal structure consistency.
• Figure 4: A Hasse diagram exemplifying a domain with a partial order in its set of classes.
• Figure 5: (a) Driving scene from the BDD100K dataset bdd100k. (b) Respective reduced mask.