LucidPPN: Unambiguous Prototypical Parts Network for User-centric Interpretable Computer Vision

TL;DR

LucidPPN tackles the ambiguity of prototypical parts by introducing a two-branch architecture that disentangles color from non-color visual features. ShapeTexNet processes grayscale inputs to capture texture/shape, while ColorNet models color, and their activations are fused to form part-based predictions aligned with semantic object parts via PDiscoNet masks. The model optimizes a triple loss L = $\alpha_D L_D + \alpha_S L_S + \alpha_C L_C$, enabling clear correspondences between prototypical parts and object parts while producing intuitive, color-aware explanations. Empirical results on four fine-grained datasets show competitive accuracy, with color disentanglement improving interpretability and robustness to color perturbations, complemented by a user study demonstrating superior human understanding compared to prior PP-based methods.

Abstract

Prototypical parts networks combine the power of deep learning with the explainability of case-based reasoning to make accurate, interpretable decisions. They follow the this looks like that reasoning, representing each prototypical part with patches from training images. However, a single image patch comprises multiple visual features, such as color, shape, and texture, making it difficult for users to identify which feature is important to the model. To reduce this ambiguity, we introduce the Lucid Prototypical Parts Network (LucidPPN), a novel prototypical parts network that separates color prototypes from other visual features. Our method employs two reasoning branches: one for non-color visual features, processing grayscale images, and another focusing solely on color information. This separation allows us to clarify whether the model's decisions are based on color, shape, or texture. Additionally, LucidPPN identifies prototypical parts corresponding to semantic parts of classified objects, making comparisons between data classes more intuitive, e.g., when two bird species might differ primarily in belly color. Our experiments demonstrate that the two branches are complementary and together achieve results comparable to baseline methods. More importantly, LucidPPN generates less ambiguous prototypical parts, enhancing user understanding.

Figures (19)

• Figure 1: Our novel prototypical parts-based model, LucidPPN, enables the disentangling of color information from the prototypical parts. This capability allows us to examine more closely the differences between an image patch and patches representing a prototypical part. As shown in the image, our model can visualize that the head of a bird, compared to the prototypical part of a bird's head from different classes, shows a high resemblance in shape and texture but differs in color. Such detailed analysis was not possible with previous prototypical parts-based approaches.
• Figure 2: Our novel type of visualization utilizes the fact that the successive prototypes in each class of LucidPPN correspond to the same object parts. That is why we use a schematic drawing of a bird to show the location of the specific prototypical parts. Moreover, LucidPPN disentangles color features from the prototypical parts to present pairs of a simplified gray prototypical part and a corresponding color. The aggregated resemblance is obtained by multiplying the resemblance to the prototypical part and the resemblance to the corresponding color.
• Figure 3: LucidPPN architecture consists of two branches: ColorNet and ShapeTexNet that encode color and shape with texture in feature maps $Z_C$ and $Z_S$, respectively. Thanks to a special type of training each channel of a feature map corresponds to similarity to a specific object part of a given class. In this image, green and orange correspond to two object parts: head and belly, and red and blue correspond to classes A and B. Therefore, each feature map consists of four channels for head of A, belly of A, head of B, and belly of B. Corresponding channels from both branches are multiplied to obtain feature map $Z_A$, which is then pooled with maximum to obtain the resemblance of prototypical parts fusion and aggregated through mean to obtain final logits.
• Figure 4: LucidPPN training schema. We use segmentation masks from PDiscoNet to align the activation of prototypical parts with object parts. Additionally, we enforce the ShapeTexNet to encode as much predictive information as possible through the usage of $L_S$. Lastly, we learn how to classify images through $L_A$ which is a binary cross-entropy loss.
• Figure 5: Influence of the number of object parts $K$ on LucidPPN accuracy. Increasing the number of parts improves the accuracy of the model. Note that each dataset is plotted in a unique color.