Multiclass ROC
Liang Wang, Luis Carvalho
TL;DR
The work tackles multiclass model evaluation by extending ROC/AUC via a binomial matrix factorization of pairwise TPR/FPR counts. It constructs matrices $M^{tp}$ and $M^{fp}$ for all class pairs, and learns a rank-1 representation that yields a ROC-like plot and an AUC-like score while accommodating misclassification costs and bootstrap-based CIs. The method is invariant to class skew and provides a straightforward visualization of classifier performance across confidence thresholds. Empirical results on simulations and real data show competitive performance against pairwise-averaged AUC and demonstrate useful uncertainty quantification.
Abstract
Model evaluation is of crucial importance in modern statistics application. The construction of ROC and calculation of AUC have been widely used for binary classification evaluation. Recent research generalizing the ROC/AUC analysis to multi-class classification has problems in at least one of the four areas: 1. failure to provide sensible plots 2. being sensitive to imbalanced data 3. unable to specify mis-classification cost and 4. unable to provide evaluation uncertainty quantification. Borrowing from a binomial matrix factorization model, we provide an evaluation metric summarizing the pair-wise multi-class True Positive Rate (TPR) and False Positive Rate (FPR) with one-dimensional vector representation. Visualization on the representation vector measures the relative speed of increment between TPR and FPR across all the classes pairs, which in turns provides a ROC plot for the multi-class counterpart. An integration over those factorized vector provides a binary AUC-equivalent summary on the classifier performance. Mis-clasification weights specification and bootstrapped confidence interval are also enabled to accommodate a variety of of evaluation criteria. To support our findings, we conducted extensive simulation studies and compared our method to the pair-wise averaged AUC statistics on benchmark datasets.