MLMC: Interactive multi-label multi-classifier evaluation without confusion matrices

TL;DR

MLMC tackles the challenge of evaluating multi-label classifiers where traditional confusion matrices scale poorly with the number of labels, by introducing an interactive visualization that supports instance-, label-, and classifier-perspective analyses. It combines a dot chart, performance bars, a precision-recall scatterplot, and a similarity matrix, all linked to enable cross-view exploration and detailed error analysis, with an emphasis on the Jaccard-based instance similarity $J_i(G_i,P_i)$ and label-wise $P_l$, $R_l$, and $F_1$ measures. The approach is grounded in clearly defined requirements (R1–R5), tasks (T1–T2 across three granularity levels), and design goals (G1–G5), and is demonstrated through three use cases (audio, image, text) alongside usability and performance studies. Findings show improved task efficiency, more consistent decision-making, and high usability (SUS ≈ 83/100), revealing MLMC’s practical value for comparative multi-label evaluation and its potential for scaling beyond traditional methods, while acknowledging limits in instance scalability and proposing future enhancements.

Abstract

Machine learning-based classifiers are commonly evaluated by metrics like accuracy, but deeper analysis is required to understand their strengths and weaknesses. MLMC is a visual exploration tool that tackles the challenge of multi-label classifier comparison and evaluation. It offers a scalable alternative to confusion matrices which are commonly used for such tasks, but don't scale well with a large number of classes or labels. Additionally, MLMC allows users to view classifier performance from an instance perspective, a label perspective, and a classifier perspective. Our user study shows that the techniques implemented by MLMC allow for a powerful multi-label classifier evaluation while preserving user friendliness.

Figures (16)

• Figure 1: Five classifiers for scientific paper labeling are compared in MLMC: (a) The label names, the ground truth (reference), and the classifier predictions are uploaded by the user. (b) Instance perspective: The paper titles, abstracts, and their label predictions by each classifier are shown in a scrollable list. (c) Classifier perspective: Label cardinality, mean F1 score, mean Precision, and mean Recall are shown for each classifier. (d) Label perspective: F1 scores for each label are represented by bars. (e) Label perspective: Interactive scatterplot shows Precision and Recall for each label and the respective mean centroids. (f) Classifier similarity matrix.
• Figure 2: Comparing the ground truth of one instance with a classifier prediction. If the label appears in both, the ground truth and the classifier prediction, it is considered true positive (TP). If it appears in neither, it is considered a true negative (TN). If the label appears in the classifier prediction, but not in the ground truth, it is considered a false positive (FP). Finally, if the label appears in the ground truth but not in the prediction, it is considered a false negative (FN).
• Figure 3: One audio file was labeled by nine different classifiers, which are represented by predictions P0 to P8. The ground truth for this audio file instance is Child and Female, which describe the recording correctly. Labels which are not in the ground truth are indicated by the grey color. FP errors are therefore made by P1, P2, and P7. P3 to P6 predict Child correctly but not Female (FN error).
• Figure 4: The F1 measure for each label is shown for each classifier within a visually augmented sortable matrix. The columns represent the classifier prediction files while the rows represent the seven different labels that can be assigned to each instance.
• Figure 5: Changing the bar orientation simplifies comparison across different classifiers for T2.2.