Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

A Deep Positive-Negative Prototype Approach to Integrated Prototypical Discriminative Learning

Ramin Zarei-Sabzevar, Ahad Harati

TL;DR

This work tackles the interpretability-accuracy trade-off in representation learning by unifying prototype-based learning with discriminative training through a shared prototype-weight representation. It introduces Deep Positive Prototype (DPP) and extends it to Deep Positive-Negative Prototype (DPNP) by incorporating negative prototypes that enforce inter-class separation, using a loss that blends cross-entropy with prototype alignment and repulsion terms. Empirical results on CIFAR-10, CIFAR-100, and Flower-102 show state-of-the-art performance with smaller networks and in reduced latent spaces, supported by angular-separation analyses and visualizations that confirm improved latent-space geometry. The approach offers improved interpretability, efficiency, and applicability to a wide range of deep architectures and domains, making it a compelling direction for robust, explainable visual recognition.

Abstract

This paper proposes a novel Deep Positive-Negative Prototype (DPNP) model that combines prototype-based learning (PbL) with discriminative methods to improve class compactness and separability in deep neural networks. While PbL traditionally emphasizes interpretability by classifying samples based on their similarity to representative prototypes, it struggles with creating optimal decision boundaries in complex scenarios. Conversely, discriminative methods effectively separate classes but often lack intuitive interpretability. Toward exploiting advantages of these two approaches, the suggested DPNP model bridges between them by unifying class prototypes with weight vectors, thereby establishing a structured latent space that enables accurate classification using interpretable prototypes alongside a properly learned feature representation. Based on this central idea of unified prototype-weight representation, Deep Positive Prototype (DPP) is formed in the latent space as a representative for each class using off-the-shelf deep networks as feature extractors. Then, rival neighboring class DPPs are treated as implicit negative prototypes with repulsive force in DPNP, which push away DPPs from each other. This helps to enhance inter-class separation without the need for any extra parameters. Hence, through a novel loss function that integrates cross-entropy, prototype alignment, and separation terms, DPNP achieves well-organized feature space geometry, maximizing intra-class compactness and inter-class margins. We show that DPNP can organize prototypes in nearly regular positions within feature space, such that it is possible to achieve competitive classification accuracy even in much lower-dimensional feature spaces. Experimental results on several datasets demonstrate that DPNP outperforms state-of-the-art models, while using smaller networks.

A Deep Positive-Negative Prototype Approach to Integrated Prototypical Discriminative Learning

TL;DR

This work tackles the interpretability-accuracy trade-off in representation learning by unifying prototype-based learning with discriminative training through a shared prototype-weight representation. It introduces Deep Positive Prototype (DPP) and extends it to Deep Positive-Negative Prototype (DPNP) by incorporating negative prototypes that enforce inter-class separation, using a loss that blends cross-entropy with prototype alignment and repulsion terms. Empirical results on CIFAR-10, CIFAR-100, and Flower-102 show state-of-the-art performance with smaller networks and in reduced latent spaces, supported by angular-separation analyses and visualizations that confirm improved latent-space geometry. The approach offers improved interpretability, efficiency, and applicability to a wide range of deep architectures and domains, making it a compelling direction for robust, explainable visual recognition.

Abstract

This paper proposes a novel Deep Positive-Negative Prototype (DPNP) model that combines prototype-based learning (PbL) with discriminative methods to improve class compactness and separability in deep neural networks. While PbL traditionally emphasizes interpretability by classifying samples based on their similarity to representative prototypes, it struggles with creating optimal decision boundaries in complex scenarios. Conversely, discriminative methods effectively separate classes but often lack intuitive interpretability. Toward exploiting advantages of these two approaches, the suggested DPNP model bridges between them by unifying class prototypes with weight vectors, thereby establishing a structured latent space that enables accurate classification using interpretable prototypes alongside a properly learned feature representation. Based on this central idea of unified prototype-weight representation, Deep Positive Prototype (DPP) is formed in the latent space as a representative for each class using off-the-shelf deep networks as feature extractors. Then, rival neighboring class DPPs are treated as implicit negative prototypes with repulsive force in DPNP, which push away DPPs from each other. This helps to enhance inter-class separation without the need for any extra parameters. Hence, through a novel loss function that integrates cross-entropy, prototype alignment, and separation terms, DPNP achieves well-organized feature space geometry, maximizing intra-class compactness and inter-class margins. We show that DPNP can organize prototypes in nearly regular positions within feature space, such that it is possible to achieve competitive classification accuracy even in much lower-dimensional feature spaces. Experimental results on several datasets demonstrate that DPNP outperforms state-of-the-art models, while using smaller networks.
Paper Structure (14 sections, 13 equations, 5 figures, 4 tables, 1 algorithm)

This paper contains 14 sections, 13 equations, 5 figures, 4 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Preliminaries
  4. Proposed method
  5. Modeling Classifier Weights and Class Centers as Shared Parameters (Prototypes)
  6. Deep Positive Prototype Model
  7. Integration of Negative Prototypes: Deep Positive-Negative Prototype Model
  8. Experiments
  9. Datasets
  10. Model Architecture and Experimental Setup
  11. Classification Accuracy
  12. Inter-Class Separability and Intra-Class Compactness
  13. Visualization of the Feature Space on CIFAR-10 Dataset
  14. Conclusion

Figures (5)

  • Figure 1: Adding discriminative terms to the loss function of deep neural networks: On the left, a commonly used approach is depicted where class centers and classifier weights are separately stored and learned. On the right, our proposed model is shown that unifies storage and learning of the weights and the centers. Here, each color represents a specific class center or the classifier weight vector associated with the corresponding neuron.
  • Figure 2: Left: The standard ResNet18 architecture with 512D feature space and about 11 million parameters. Right: The reduced ResNet18 architecture with 256 filters in the final convolutional layer, bottlenecked at a (3D or 10D, based on the dataset) feature space and about 5 million parameters. This reduced architecture is used to study the regularity of feature space.
  • Figure 3: Histograms of inter-class angles for different methods: the first two rows correspond to CIFAR-10, and the bottom two rows show the results of CIFAR-100. Standard ResNet-18 architecture is used in odd rows, while reduced ResNet-18 is used in even rows. The proposed models show concentrated histograms with larger minimums which denote better separation.
  • Figure 4: Histogram of intra-class angles denotes the resulted compactness from different methods: the first two rows correspond to CIFAR-10 and the bottom two rows to CIFAR-100. Standard ResNet-18 architecture is used in odd rows, while reduced ResNet-18 is used in even rows. The proposed models show concentrated histograms similar to CL, CCL, and SCCL which all use center pulling terms in the loss function to obtain better compactness.
  • Figure 5: Visualization of the feature space obtained by different models for CIFAR-10 using CE, CL, CCL, SCCL, CPN and the proposed DPP and DPNP models. Sub-figures (a)–(g) present 2D angular histograms of spherical coordinates $\phi$ and $\theta$ of data points after length normalization, mapping the surface of a unit-radius sphere onto each plot. Class centers are marked by red circles and weight vectors are depicted as blue circles. Sub-figure (h) retains the original 3D feature space visualization for the DPNP model, showing the spatial distribution of class members and their clear separation. The encircled classes are located behind while the others face the current view.