Multi-Scale Grouped Prototypes for Interpretable Semantic Segmentation

TL;DR

ScaleProtoSeg tackles interpretable semantic segmentation by learning multi-scale prototypical parts and enforcing sparse cross-scale groupings, exposing how details at different receptive fields contribute to pixel-level predictions. The method combines a multi-scale prototype layer with a class-specific grouping mechanism, optimized in two stages to promote diversity and sparsity while preserving accuracy. Across Pascal VOC, Cityscapes, and ADE20K, ScaleProtoSeg outperforms ProtoSeg in interpretability metrics and achieves competitive or superior mIoU, especially on complex scenes with objects at multiple depths. The approach offers actionable, visualizable explanations via grouped prototype activations and demonstrates transferability to new domains and large datasets with reduced computational overhead relative to prior prototype-based methods.

Abstract

Prototypical part learning is emerging as a promising approach for making semantic segmentation interpretable. The model selects real patches seen during training as prototypes and constructs the dense prediction map based on the similarity between parts of the test image and the prototypes. This improves interpretability since the user can inspect the link between the predicted output and the patterns learned by the model in terms of prototypical information. In this paper, we propose a method for interpretable semantic segmentation that leverages multi-scale image representation for prototypical part learning. First, we introduce a prototype layer that explicitly learns diverse prototypical parts at several scales, leading to multi-scale representations in the prototype activation output. Then, we propose a sparse grouping mechanism that produces multi-scale sparse groups of these scale-specific prototypical parts. This provides a deeper understanding of the interactions between multi-scale object representations while enhancing the interpretability of the segmentation model. The experiments conducted on Pascal VOC, Cityscapes, and ADE20K demonstrate that the proposed method increases model sparsity, improves interpretability over existing prototype-based methods, and narrows the performance gap with the non-interpretable counterpart models. Code is available at github.com/eceo-epfl/ScaleProtoSeg.

Figures (35)

• Figure 1: ScaleProtoSeg learns scale-specific prototypes at multiple scales and a sparse prototype grouping to extract patterns referring to different levels of details or scales.
• Figure 2: Overall architecture of ScaleProtoSeg. Each color in the feature maps and following layers corresponds to a specific scale ($S=4$ and $M = 3$ in this illustration).
• Figure 3: Analysis of the binarized prototype activations at multiple percentile thresholds $p_{th} \in \{0.8, 0.9, 0.99\}$ on Cityscapes and ADE20K validation sets.
• Figure 4: ScaleProtoSeg provides the interpretation of a segmentation through the analysis of groups of prototypes. For the example of the class car on Cityscapes, $2$ prototypes per scale (whose activations are displayed at the top of the figure) are used by the model across the $3$ learned groups shown at the bottom right. For this class, groups correspond to the bottom part, the main part or the upper part of the car.
• Figure 5: Model prototype and group assignments for the class bus, bed, and horse on Cityscapes, ADE20K, and Pascal.