Framework-agnostic Semantically-aware Global Reasoning for Segmentation

TL;DR

SGR tackles the lack of scene semantics in global reasoning for segmentation by learning $K$ latent semantic tokens $T$ projected from features via $K$ concept regions $P$, refined by a Transformer encoder, and re-projected to boost pixel features. The method uses weak supervision through connected components of ground-truth masks and novel losses plus cosine diversity constraints to enforce disjoint, semantic concept regions. It introduces interpretability metrics—class-semantics $\mathcal{S}_C$, instance-semantics $\mathcal{S}_I$, and token-diversity $\mathcal{D}_C$, $\mathcal{D}_I$—to quantify semantic richness and diversity of latent tokens. Across Cityscapes, ADE-20K, and COCO-Stuffs-10K, SGR yields consistent improvements across backbones and heads and transfers to downstream tasks like object detection, demonstrating that semantically meaningful latent representations enhance both segmentation and broader visual understanding.

Abstract

Recent advances in pixel-level tasks (e.g. segmentation) illustrate the benefit of of long-range interactions between aggregated region-based representations that can enhance local features. However, such aggregated representations, often in the form of attention, fail to model the underlying semantics of the scene (e.g. individual objects and, by extension, their interactions). In this work, we address the issue by proposing a component that learns to project image features into latent representations and reason between them using a transformer encoder to generate contextualized and scene-consistent representations which are fused with original image features. Our design encourages the latent regions to represent semantic concepts by ensuring that the activated regions are spatially disjoint and the union of such regions corresponds to a connected object segment. The proposed semantic global reasoning (SGR) component is end-to-end trainable and can be easily added to a wide variety of backbones (CNN or transformer-based) and segmentation heads (per-pixel or mask classification) to consistently improve the segmentation results on different datasets. In addition, our latent tokens are semantically interpretable and diverse and provide a rich set of features that can be transferred to downstream tasks like object detection and segmentation, with improved performance. Furthermore, we also proposed metrics to quantify the semantics of latent tokens at both class \& instance level.

Figures (9)

• Figure 1: SGR learns to group pixels into latent tokens based on semantic similarity (top); the tokens are refined using a Transformer and back-projected to enhance the original features for segmentation. Notably, the resulting tokens are more semantic than prior work, as measured by the proposed class-semantics ($S_C$) and instance-semantics ($S_I$) metrics ($\downarrow$ is better), and are more diverse ($\uparrow$ is better), as measured by proposed diversity metric at class($D_C$) and instance-level($D_I$) (see Maskformer cheng2021per in the bottom).
• Figure 2: Overview of our framework. A dot-product is performed between $K$ learned concept embeddings (randomly initialized) and position-aware image features with added x-y positional embedding to generate soft projection matrices ( concept regions). The projection matrices aggregate features in an object-centric manner to create latent tokens. A two-layer transformer encoder is used to reason between the tokens which are re-projected to original feature space for segmentation.
• Figure 3: Visualization of our greedy matching strategy. Binary mask losses are used to compute a cost matrix, based on which, Hungarian matching is applied to perform 1-to-1 matching between the concept regions and the ground truth components. Next, we greedily select $L$-top matches from the remaining ones. Once matched, in training, we compute losses between the union of predicted masks assigned to the same connected component.
• Figure 4: Qualitative results showing that our SGR component generates more semantically meaningful and diverse tokens. In all three images, SGR was able to disambiguate between the instances (was able to differentiate the cow in the left in the first image, and different groups/ instances of the car in the last row) unlike Maskformer cheng2021per. GCNET chen2019graph tokens, on the other hand, lack strong semantic meaning.
• Figure A: Ground truth segmentation mask and corresponding connected components. The connected components are assumed to give a lower bound on the number of instances; e.g., the illustrated image contains 6 people but the number of connected components that corresponds to person is only 3.