ProtoQuant: Quantization of Prototypical Parts For General and Fine-Grained Image Classification

TL;DR

ProtoQuant addresses the interpretability-explainability gap in large-scale image classification by grounding prototypical parts in a discrete concept codebook learned with a Vector-Quantized Variational Autoencoder (VQ-VAE) style mechanism while keeping the backbone frozen. It replaces the traditional classifier head with a modular concept-based head, comprising concept matching, activation mapping, and a nonnegative concept-to-class mapping, trained in two stages to ensure stability and grounding. The approach achieves robust prototype stability against perturbations and demonstrates competitive accuracy on ImageNet and strong performance on fine-grained benchmarks, all while enabling efficient model editing and pruning. By tying predictions to a finite vocabulary of human-understandable concepts, ProtoQuant provides reliable, localized explanations suitable for auditing and deployment in high-stakes settings, with clear improvements in scalability and interpretability over prior prototypical-part methods.

Abstract

Prototypical parts-based models offer a "this looks like that" paradigm for intrinsic interpretability, yet they typically struggle with ImageNet-scale generalization and often require computationally expensive backbone finetuning. Furthermore, existing methods frequently suffer from "prototype drift," where learned prototypes lack tangible grounding in the training distribution and change their activation under small perturbations. We present ProtoQuant, a novel architecture that achieves prototype stability and grounded interpretability through latent vector quantization. By constraining prototypes to a discrete learned codebook within the latent space, we ensure they remain faithful representations of the training data without the need to update the backbone. This design allows ProtoQuant to function as an efficient, interpretable head that scales to large-scale datasets. We evaluate ProtoQuant on ImageNet and several fine-grained benchmarks (CUB-200, Cars-196). Our results demonstrate that ProtoQuant achieves competitive classification accuracy while generalizing to ImageNet and comparable interpretability metrics to other prototypical-parts-based methods.

Figures (19)

• Figure 1: Standard ProtoPNet-based models (e.g. PIP-Net) exhibit representational instability: modifying a local region of an image can cause a drastic shift in the highest-activating prototypes, even if the modification is in a non-relevant area. Contrary, ProtoQuant maintains stable activations despite input modifications.
• Figure 2: Architecture of ProtoQuant. Our model consists of a frozen backbone $\phi$ that produces feature map $\phi(x)$ from input image $x$. Then each feature vector $z_{ij}$ is compared with prototypical parts from learned codebook $C$ through a mapping function $m$. As a result, we obtain concept activation maps $p$ over which we take the maximum value to obtain concept presence scores $s$, where $s_m$ corresponds to the probability of a presence of code $c_m$. Lastly, those score are multiplied with classification matrix $W$ to obtain final logit values.
• Figure 3: Training of ProtoQuant is in two stages. The first one focuses on training a codebook to obtain meaningful prototypical parts, while the second switches original head with sparse and non-negative classification matrix to ensure interpretability.
• Figure 4: Visualization of the top 3 activated prototypes from ProtoQuant on an example from CUB-200.
• Figure 5: Representative prototypical explanations on ImageNet. The input (daisy) is shown in the first column, with rows visualizing the most important prototypical parts via their nearest neighbors in the training set (columns 2–5). The activation of patches from multiple distinct classes for a single prototype highlights the capacity of ProtoQuant for inter-class concept sharing, where visual primitives are leveraged as a universal basis for classification.