StackGenVis: Alignment of Data, Algorithms, and Models for Stacking Ensemble Learning Using Performance Metrics

TL;DR

StackGenVis tackles the complexity of constructing stacking ensembles by introducing a knowledge generation model and a visual analytics system that align data, algorithms, and models using multiple performance metrics. The approach enables metric-driven exploration, data wrangling, feature selection, and metamodel comparison, reducing trial-and-error and increasing interpretability. Two real-world use cases (healthcare and sentiment/stance detection) show performance gains (≈88% accuracy on heart disease) and robust cross-validation results, supported by expert interviews. The work provides a modular, provenance-rich workflow with multiple coordinated views and JSON-exportable stacking solutions, offering practical impact for practitioners seeking trustworthy, efficient stacked generalization.

Abstract

In machine learning (ML), ensemble methods such as bagging, boosting, and stacking are widely-established approaches that regularly achieve top-notch predictive performance. Stacking (also called "stacked generalization") is an ensemble method that combines heterogeneous base models, arranged in at least one layer, and then employs another metamodel to summarize the predictions of those models. Although it may be a highly-effective approach for increasing the predictive performance of ML, generating a stack of models from scratch can be a cumbersome trial-and-error process. This challenge stems from the enormous space of available solutions, with different sets of data instances and features that could be used for training, several algorithms to choose from, and instantiations of these algorithms using diverse parameters (i.e., models) that perform differently according to various metrics. In this work, we present a knowledge generation model, which supports ensemble learning with the use of visualization, and a visual analytics system for stacked generalization. Our system, StackGenVis, assists users in dynamically adapting performance metrics, managing data instances, selecting the most important features for a given data set, choosing a set of top-performant and diverse algorithms, and measuring the predictive performance. In consequence, our proposed tool helps users to decide between distinct models and to reduce the complexity of the resulting stack by removing overpromising and underperforming models. The applicability and effectiveness of StackGenVis are demonstrated with two use cases: a real-world healthcare data set and a collection of data related to sentiment/stance detection in texts. Finally, the tool has been evaluated through interviews with three ML experts.

Figures (7)

• Figure 1: Knowledge generation model for ensemble learning with VA derived from the model by Sacha et al. Sacha2014Knowledge. On the left, it illustrates how a VA system can enable the exploration of the data and the models with the use of visualization. On the right, a number of design goals assist the human in the exploration, verification, and knowledge generation for ensemble learning.
• Figure 2: The exploration process of ML algorithms. View (a.1) summarizes the performance of all available algorithms, and (a.2) the per-class performance based on precision, recall, and f1-score for each algorithm. (b) presents a selection of parameters for KNN in order to boost the per-class performance shown in (c.1). (c.2) illustrates in light blue the selected models and in gray the remaining ones. Also from (a.2), both RF and ExtraT performances seem to be equal. However in (d), after resetting class optimization, ExtraT models appear to perform better overall. In view (e), the boxplots were replaced by point clouds that represent the individual models of activated algorithms. The color encoding is the same as for the algorithms, but unselected models are greyed out. Finally, the radar chart in (f) displays a portion of the models' space in black that will be used to create the initial stack against the entire exploration space in yellow. The chart axes are normalized from 0 to 100%.
• Figure 3: The data space projection with the importance of each instance measured by the accuracy achieved by the stack models (a). The parallel coordinates plot view for the exploration of the values of the features (b); a problematic case is highlighted in red with values being null ('4' has no meaning for Ca). (c.1) shows the brushed instance from the selection in (b) and (c.2) a problematic point that causes troubles to the stacking ensemble. (c.3) indicates the various functionalities that StackGenVis is able to perform for instances.
• Figure 4: Our feature selection view that provides three different feature selection techniques. The y-axis of the table heatmap depicts the data set's features, and the x-axis depicts the selected models in the current stored stack. Univariate-, permutation-, and accuracy-based feature selection is available as long with any combination of them (a). (b) displays the normalized importance color legend. The per-model feature accuracy is depicted in (c), and (d) presents the user's interaction to disable specific features to be used for all the models (only seven features are shown here). This could also happen on an individual basis for every model.
• Figure 5: Visual exploration of the models' space. The same MDS projection is observable in varying stages with different legend ranges and diverse colors for each instance, depending on the selected performance metric. The three steps in this figure demonstrate that we can reach both performant base models but also diverse algorithms by exploration of different validation metrics in (a) and (b). With the removal of the unselected models in (c), the performance remains stable but the complexity of the stacking ensemble reduces as more models leave the previous stack (cf. \ref{['fig:teaser']}(b, Ⓢ6)).