Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Deformable ProtoPNet: An Interpretable Image Classifier Using Deformable Prototypes

Jon Donnelly, Alina Jade Barnett, Chaofan Chen

TL;DR

Deformable ProtoPNet addresses the limitation of rigid prototypes in case-based interpretable image classifiers by introducing deformable prototypes composed of prototypical parts with learnable spatial offsets. Prototypes and image features are normalized to a fixed length and matched via cosine similarity on a hypersphere, reinforced by angular-margin losses and an orthogonality regularizer to encourage discriminability and diversity among parts. Empirical results on fine-grained datasets (e.g., CUB-200-2011 and Stanford Dogs) show competitive or state-of-the-art accuracy, especially when ensembling, while providing richer explanations that align image patches with deformable prototypical parts. The work also provides a norm-preserving interpolation mechanism for fractional offsets and a differentiable path for backpropagation through deformable prototypes, enabling end-to-end training and interpretability-driven debugging.

Abstract

We present a deformable prototypical part network (Deformable ProtoPNet), an interpretable image classifier that integrates the power of deep learning and the interpretability of case-based reasoning. This model classifies input images by comparing them with prototypes learned during training, yielding explanations in the form of "this looks like that." However, while previous methods use spatially rigid prototypes, we address this shortcoming by proposing spatially flexible prototypes. Each prototype is made up of several prototypical parts that adaptively change their relative spatial positions depending on the input image. Consequently, a Deformable ProtoPNet can explicitly capture pose variations and context, improving both model accuracy and the richness of explanations provided. Compared to other case-based interpretable models using prototypes, our approach achieves state-of-the-art accuracy and gives an explanation with greater context. The code is available at https://github.com/jdonnelly36/Deformable-ProtoPNet.

Deformable ProtoPNet: An Interpretable Image Classifier Using Deformable Prototypes

TL;DR

Deformable ProtoPNet addresses the limitation of rigid prototypes in case-based interpretable image classifiers by introducing deformable prototypes composed of prototypical parts with learnable spatial offsets. Prototypes and image features are normalized to a fixed length and matched via cosine similarity on a hypersphere, reinforced by angular-margin losses and an orthogonality regularizer to encourage discriminability and diversity among parts. Empirical results on fine-grained datasets (e.g., CUB-200-2011 and Stanford Dogs) show competitive or state-of-the-art accuracy, especially when ensembling, while providing richer explanations that align image patches with deformable prototypical parts. The work also provides a norm-preserving interpolation mechanism for fractional offsets and a differentiable path for backpropagation through deformable prototypes, enabling end-to-end training and interpretability-driven debugging.

Abstract

We present a deformable prototypical part network (Deformable ProtoPNet), an interpretable image classifier that integrates the power of deep learning and the interpretability of case-based reasoning. This model classifies input images by comparing them with prototypes learned during training, yielding explanations in the form of "this looks like that." However, while previous methods use spatially rigid prototypes, we address this shortcoming by proposing spatially flexible prototypes. Each prototype is made up of several prototypical parts that adaptively change their relative spatial positions depending on the input image. Consequently, a Deformable ProtoPNet can explicitly capture pose variations and context, improving both model accuracy and the richness of explanations provided. Compared to other case-based interpretable models using prototypes, our approach achieves state-of-the-art accuracy and gives an explanation with greater context. The code is available at https://github.com/jdonnelly36/Deformable-ProtoPNet.
Paper Structure (20 sections, 43 equations, 14 figures, 5 tables)

This paper contains 20 sections, 43 equations, 14 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Deformable Prototypes
  4. Overview of a Deformable Prototype
  5. Offset Generation and Feature Interpolation
  6. Deformable ProtoPNet
  7. Training
  8. Prototype Visualizations
  9. Reasoning Process
  10. Experiments and Numerical Results
  11. Conclusion
  12. Proof of Theorem 3.1
  13. Backpropagation through a Deformable Prototype
  14. More Examples of Reasoning Processes
  15. Local Analysis: Visualizations of Most Similar Prototypes to Given Images
  16. ...and 5 more sections

Figures (14)

  • Figure 1: How an input image of a painted bunting is compared with (a) a regular (non-deformable) prototype and (b) a deformable prototype of the painted bunting class (overlaid on its source image).
  • Figure 2: How a deformable prototype is applied to the latent representation of an input image of a painted bunting. (a) The latent input $\mathbf{\hat{z}}$ is fed into the offset prediction function $\delta$ to produce (b) a field of offsets. These offsets are used to (c) alter the spatial position of each prototypical part, which are (d) compared to the input to (e) compute prototype similarity according to equation (\ref{['eq:deform_similarity']}).
  • Figure 3: The architecture for Deformable ProtoPNet.
  • Figure 4: A representation of the latent space learned by Deformable PrototPNet.
  • Figure 5: The reasoning process of a Deformable ProtoPNet with $2 \times 2$ deformable prototypes.
  • ...and 9 more figures

Theorems & Definitions (1)

  • proof