HardVis: Visual Analytics to Handle Instance Hardness Using Undersampling and Oversampling Techniques

TL;DR

The paper addresses the challenge of instance hardness in imbalanced classification and proposes a visual analytics system, HardVis, to manage both undersampling and oversampling by interactively identifying and sampling SBRO data types. It integrates multiple coordinated views (UMAP projections, box plots, inverse polar chart, Sankey tracker) and algorithmic components (OSS, NCR, SMOTE, ADASYN, XGBoost) to allow local and global sampling decisions with performance validation on a held-out test data set. The authors provide a running iris demonstration, use cases with breast cancer and vehicle datasets, and expert interviews to validate usefulness and limitations. The work contributes a coherent VA workflow for data-level balancing, a prototype implementation, and qualitative insights into user trust and practical impact in data-centric ML contexts.

Abstract

Despite the tremendous advances in machine learning (ML), training with imbalanced data still poses challenges in many real-world applications. Among a series of diverse techniques to solve this problem, sampling algorithms are regarded as an efficient solution. However, the problem is more fundamental, with many works emphasizing the importance of instance hardness. This issue refers to the significance of managing unsafe or potentially noisy instances that are more likely to be misclassified and serve as the root cause of poor classification performance. This paper introduces HardVis, a visual analytics system designed to handle instance hardness mainly in imbalanced classification scenarios. Our proposed system assists users in visually comparing different distributions of data types, selecting types of instances based on local characteristics that will later be affected by the active sampling method, and validating which suggestions from undersampling or oversampling techniques are beneficial for the ML model. Additionally, rather than uniformly undersampling/oversampling a specific class, we allow users to find and sample easy and difficult to classify training instances from all classes. Users can explore subsets of data from different perspectives to decide all those parameters, while HardVis keeps track of their steps and evaluates the model's predictive performance in a test set separately. The end result is a well-balanced data set that boosts the predictive power of the ML model. The efficacy and effectiveness of HardVis are demonstrated with a hypothetical usage scenario and a use case. Finally, we also look at how useful our system is based on feedback we received from ML experts.

Figures (6)

• Figure 1: Undersampling and oversampling certain data types with HardVis: (a) the panel with many tunable parameters for UMAP, undersampling, and oversampling; (b) box plots for comparing the values of all points against the algorithm's suggestion in each feature; (c) a stacked bar chart showing the base vs. the new distribution if the suggestion is approved; (d) a table heatmap view for comparing the instances' values across all features; (e) a UMAP projection emphasizing the additions/deletions of points, along with the data type for every instance; (f) an inverse polar chart with chords that depicts the predicted probabilities, as well as the training confusion; (g) a Sankey diagram for tracking any undersampling or oversampling confirmed actions; and (h) a visual embedding based on (e) to highlight the confusing test instances, and a horizontal bar chart to illustrate the performance difference for each step.
• Figure 2: The HardVis workflow starts by classifying the training data into four types according to the user's visual inspection of 9 alternative projections. The data is sent for either undersampling or oversampling, which can make suggestions continuously. The user's confirmation is requested after the exploratory data analysis through the visualizations.
• Figure 3: At first, a comparison of different data types projections and then two consecutive undersampling phases with the NCR algorithm are shown in this arrangement of screenshots. The default value for the number of neighbors is 5 (see (a)), which is used as input for computing the type of each instance with KNN. The projections are generated by systematically tweaking the above parameter, as illustrated in (b); the best choice is theoretically the highest value for the Shepard diagram correlation (SDC) metric. In (c), we have activated the algorithm, and we check the impact of this automated technique on the projection in (d). (e) presents the difference in distributions of all data types per class label from when the algorithm was inactive as opposed to its activation. In (f), we explore a specific rare case under removal consideration. This instance is contrasted against the remaining points of this same class (i.e., virginica in orange color); the selection was made using a lasso interaction, as demonstrated in (d). While the values for all features are lower for this sample than the rest, sepal_l appears the furthest away. Additional details can be found in (g) that highlights these differences in values of particular features and confirms our findings from the data overview. Consequently, we choose to delete this instance because it might cause further confusion to the model, as depicted in (d). The second time we deploy NCR (cf. (h)), two safe instances are in our focus since they are easily classified due to the high predicted probability visible from the inverse polar chart in (i). Therefore, we decide to remove these two points.
• Figure 4: An oversampling phase that aims to balance the data set again. According to (a), we use ADASYN for the minority class (versicolor in blue) that contains fewer instances. Also, we exclude from the input of the algorithm two rare cases near the borders of two classes, as illustrated in (b). The system proposes two artificially-created points for addition that we approve, see (c). The Sankey diagram in (d) summarizes the core execution phases with undersampling and oversampling steps. Only one test instance is confused according to (e), while the manual decisions (step 2, step 4, and step 7/8) improved the balanced accuracy and f1-score scores compared to the automated methods (steps 1, 3, and 5).
• Figure 5: The investigation of diverse structures of data types and alternative suggestions in an undersampling scenario. View (a) shows the selection of the number of neighbors value of 13, which has 75.21% Shepard diagram correlation (SDC) score, as illustrated in (b). The UMAP visible in (c) has one rare sample and 6 outliers belonging to the benign class that holds relatively normal values compared to the malignant cluster (C), as shown in (d). Therefore the suggestions for removal in C1 are valid since even humans cannot understand why these points are benign cases. On the other hand, C2 contains five rare examples and two outliers that serve as a bridge between the two classes, cf. (c). Interestingly, the three most important features differentiate the right group of points (IDs: 64, 102, and 134) from the left, i.e., size_un, shape_un, and bare_nuc in (e). This diversity is crucial when predicting difficult to classify instances, hence the analyst chooses to keep this cluster despite the NCR algorithm's suggestion for removal. C3 is the final selection, with most outliers being removed because the model badly predicted them, as seen in (f). This leads to (g), which presents an improved performance with 6 confused test instances that are cancer-free but predicted as the opposite. The malignant class is secure due to the rare cases being intact.