QPM: Discrete Optimization for Globally Interpretable Image Classification

TL;DR

QPM addresses global interpretability in image classification by learning a compact, contrastive class representation: each class is described by a binary assignment of a fixed number of features (typically $k=5$) that are shared across classes. A binary quadratic program over feature selection $\mathbf{s}$ and class-feature assignments $\mathbf{W}$ optimizes a composite objective $Z = Z_A + Z_R + Z_B$, where $Z_A$ encodes class–feature similarity, $Z_R$ penalizes feature redundancy, and $Z_B$ injects steerable biases, with constraints enforcing equal feature budgets per class and unique associations. After solving for $\mathbf{W}^*$, the features are fine-tuned to align with the assigned classes, producing faithful global explanations and unprecedented structural grounding while maintaining competitive accuracy across datasets. The approach is backbone-agnostic, provides measurable interpretability gains (e.g., SID@5, Class-Independence, Structural Grounding), and supports steerability to tailor feature selection to domain criteria.

Abstract

Understanding the classifications of deep neural networks, e.g. used in safety-critical situations, is becoming increasingly important. While recent models can locally explain a single decision, to provide a faithful global explanation about an accurate model's general behavior is a more challenging open task. Towards that goal, we introduce the Quadratic Programming Enhanced Model (QPM), which learns globally interpretable class representations. QPM represents every class with a binary assignment of very few, typically 5, features, that are also assigned to other classes, ensuring easily comparable contrastive class representations. This compact binary assignment is found using discrete optimization based on predefined similarity measures and interpretability constraints. The resulting optimal assignment is used to fine-tune the diverse features, so that each of them becomes the shared general concept between the assigned classes. Extensive evaluations show that QPM delivers unprecedented global interpretability across small and large-scale datasets while setting the state of the art for the accuracy of interpretable models.

Figures (34)

• Figure 1: Faithful global interpretability of our : Without any additional supervision, learns to represent Rottweiler and Doberman using $5$ diverse and general features. faithfully explains that it differentiates them exclusively via their visibly distinct head.
• Figure 2: Exemplary Application of the QP to a dense model with just 4 features and 2 classes with the aim of selecting 3 features and assigning 2 per class. The different weights are indicated by thickness of connection and color indicates sign. The result is a binary assignment of selected features to classes. Typical values for on are selecting 50 features out of 2048 and assigning 5 to each of the 200 classes.
• Figure 3: Overview of our proposed pipeline to construct a
• Figure 4: Contrastive faithful class explanations for trained on : Without any additional supervision, learns to differentiate Shiny and Bronzed Cowbird ($\mathrm{\Psi}^{gt}=0.97$) using the red eye just like humans do, as the annotations in or the screenshot in \ref{['fig:ScreenshotDiffShiny']} show.
• Figure 5: Extreme Examples for feature distributions and their Contrastiveness on .