VisRuler: Visual Analytics for Extracting Decision Rules from Bagged and Boosted Decision Trees

TL;DR

Bagging and boosting ensembles such as Random Forest (RF) and AdaBoost (AB) offer strong predictive performance but pose interpretability challenges due to numerous decision paths. VisRuler introduces a visual analytics workflow and a prototype tool that extracts, compares, and reason about decision rules across RF and AB, enabling collaboration between ML experts and domain experts. The system provides five coordinated views for model selection, global feature ranking, decisions exploration, manual decision extraction, and decision evaluation, validated through a real-world use case and a user study. By surfacing robust, explainable rules from diverse ensembles, VisRuler enhances transparency and trust for high-stakes decisions in domains like finance, healthcare, and social care.

Abstract

Bagging and boosting are two popular ensemble methods in machine learning (ML) that produce many individual decision trees. Due to the inherent ensemble characteristic of these methods, they typically outperform single decision trees or other ML models in predictive performance. However, numerous decision paths are generated for each decision tree, increasing the overall complexity of the model and hindering its use in domains that require trustworthy and explainable decisions, such as finance, social care, and health care. Thus, the interpretability of bagging and boosting algorithms, such as random forest and adaptive boosting, reduces as the number of decisions rises. In this paper, we propose a visual analytics tool that aims to assist users in extracting decisions from such ML models via a thorough visual inspection workflow that includes selecting a set of robust and diverse models (originating from different ensemble learning algorithms), choosing important features according to their global contribution, and deciding which decisions are essential for global explanation (or locally, for specific cases). The outcome is a final decision based on the class agreement of several models and the explored manual decisions exported by users. We evaluated the applicability and effectiveness of VisRuler via a use case, a usage scenario, and a user study. The evaluation revealed that most users managed to successfully use our system to explore decision rules visually, performing the proposed tasks and answering the given questions in a satisfying way.

Figures (10)

• Figure 1: Extracting decision rules for manual evaluation with VisRuler: (a) panel with visual metaphors for selecting performant and diverse models; (b) box plot for feature selection according to per algorithmic importance; (c) visual embedding of computed decisions that training instances fall in due to their values; (d) vertical parallel coordinates plot that summarizes the rules with value ranges for each feature and highlights the current test instance; and (e) horizontal stacked bar chart for revealing the class agreement of each model against the manual decisions, together with the parallel coordinates plots for tuning hyperparameters and training new models.
• Figure 2: The VisRuler workflow allows ML experts to select performant and diverse models, choose important features, and retrain models with new hyperparameters. Domain experts can explore robust decisions, compare them to global standards, identify local decisions for a specific test instance, and extract them.
• Figure 3: The VisRuler cooperation diagram illustrates how synchronous co-located collaboration typically happens between the ML expert and the domain expert. Three phases and five panels support their teamwork in a single-page tool, with the ML expert being more active (orange color) than the domain expert who receives and analyzes information (teal color). The seven linear steps taken by each user are also noted at the bottom.
• Figure 4: Exploration of ML models with VisRuler. View (a) presents the deactivation of all models except for RF8, RF10, and AB10, after consideration of their performance based on multiple metrics displayed in the visualizations. In (b), Generosity is the least important feature for the three active ML models and, particularly, its importance decreased while we deactivated most of the available ML models (see brown color). View (c) indicates that, after retraining with 5 of 6 original features, the new AB8 is better than the subsequent models due to the decline in recall; AB8, RF9, and RF10 remain the only active models after this step. In the box plot (d), the feature H life exp becomes more important by far than GDP per cap. Thus, these features swapped places compared to view (b).
• Figure 5: Examining several pure global decisions from the active AB model. In (a), we activate the density view in order to distinguish where most decisions are positioned. Note that this screenshot is composed of the decisions space (DS) view and the settings for the same view plus the settings for the manual decisions (MD) view. In (b), we select step-by-step three clusters of 12 identical decisions each. The decisions for Ⓒ1 classify training instances only for HS-Level-3 class (as depicted in (c)). Similarly, Ⓒ2 contains decisions for HS-Level-2 (visible in (d)), while Ⓒ3 for the remaining class, as shown in (e). The 7 test instance, which is currently under investigation, cannot be classified by those prior decisions. However, it most likely belongs in the medium- or the high-level class.