Silhouettes and quasi residual plots for neural nets and tree-based classifiers

TL;DR

The paper addresses understanding why classifiers misclassify by visualizing classification cases rather than just model mechanics. It introduces PAC-based silhouette plots, quasi residual plots, and class maps to diagnose label bias, class overlap, and data-driven factors affecting prediction accuracy, with formal definitions for $\mathrm{PAC}(i)$, $s(i)$, and $\mathrm{farness}$. The methods are demonstrated on CIFAR-10 with a ResNet-56, Titanic data with CART, and emotion tweets with random forests, revealing when misclassifications stem from feature noise versus label noise. The approach yields per-case diagnostics and cross-class comparisons, and is supported by an R package (classmapclassmap) for broader applicability to other classifiers.

Abstract

Classification by neural nets and by tree-based methods are powerful tools of machine learning. There exist interesting visualizations of the inner workings of these and other classifiers. Here we pursue a different goal, which is to visualize the cases being classified, either in training data or in test data. An important aspect is whether a case has been classified to its given class (label) or whether the classifier wants to assign it to different class. This is reflected in the (conditional and posterior) probability of the alternative class (PAC). A high PAC indicates label bias, i.e. the possibility that the case was mislabeled. The PAC is used to construct a silhouette plot which is similar in spirit to the silhouette plot for cluster analysis (Rousseeuw, 1987). The average silhouette width can be used to compare different classifications of the same dataset. We will also draw quasi residual plots of the PAC versus a data feature, which may lead to more insight in the data. One of these data features is how far each case lies from its given class. The graphical displays are illustrated and interpreted on benchmark data sets containing images, mixed features, and tweets.

Figures (21)

• Figure 1: Silhouette plots of (left) a partitioning of a toy dataset in three clusters, and (right) a classification with four classes.
• Figure 2: Quasi residual plots of images versus their intensity and hue.
• Figure 3: Silhouette plot on the CIFAR-10 test data.
• Figure 4: Class map of the airplane class in the test data, with the images corresponding to the marked points.
• Figure 5: Class map of the cat class in the test data, with the corresponding images.