FeatureEnVi: Visual Analytics for Feature Engineering Using Stepwise Selection and Semi-Automatic Extraction Approaches

TL;DR

FeatureEnVi tackles the challenge of feature engineering in ML by delivering a visual analytics workflow that unifies stepwise feature selection with semi-automatic feature extraction. The system partitions data into four probability-based slices, provides multiple automatic feature-selection techniques, and offers rich visualizations (data space, radial tree, graph, punchcard) to reason about feature transformations and generations. Across use cases and a case study, FeatureEnVi demonstrates improved predictive performance with fewer engineered features and gains in interpretability, complemented by qualitative feedback from ML and VA experts. The work contributes a cohesive VA framework for feature engineering that can enhance transparency, efficiency, and trust in ML models, with clear pathways for scalability and extension to other data types and advanced analyses.

Abstract

The machine learning (ML) life cycle involves a series of iterative steps, from the effective gathering and preparation of the data, including complex feature engineering processes, to the presentation and improvement of results, with various algorithms to choose from in every step. Feature engineering in particular can be very beneficial for ML, leading to numerous improvements such as boosting the predictive results, decreasing computational times, reducing excessive noise, and increasing the transparency behind the decisions taken during the training. Despite that, while several visual analytics tools exist to monitor and control the different stages of the ML life cycle (especially those related to data and algorithms), feature engineering support remains inadequate. In this paper, we present FeatureEnVi, a visual analytics system specifically designed to assist with the feature engineering process. Our proposed system helps users to choose the most important feature, to transform the original features into powerful alternatives, and to experiment with different feature generation combinations. Additionally, data space slicing allows users to explore the impact of features on both local and global scales. FeatureEnVi utilizes multiple automatic feature selection techniques; furthermore, it visually guides users with statistical evidence about the influence of each feature (or subsets of features). The final outcome is the extraction of heavily engineered features, evaluated by multiple validation metrics. The usefulness and applicability of FeatureEnVi are demonstrated with two use cases and a case study. We also report feedback from interviews with two ML experts and a visualization researcher who assessed the effectiveness of our system.

Figures (7)

• Figure 1: Selecting important features, transforming them, and generating new features with FeatureEnVi: (a) the horizontal beeswarm plot for manually slicing the data space (which is sorted by predicted probabilities) and continuously checking the migration of data instances throughout the process; (b) the table heatmap view for the selection of features according to feature importances calculated from automatic techniques; (c) the radial tree providing an overview of the features with statistical measures for the different groups of instances, as set by the user-defined data slices; (d) the graph visualization for the detailed exploration of features, their transformation, and comparison between two or three features for feature generation purposes; and (e) the punchcard for tracking the steps of the process and the grouped bar chart for comparing the current vs. the best predictive performance based on three validation metrics.
• Figure 2: The FeatureEnVi workflow begins with partitioning the data set according to the prediction probabilities of instances. The data is passed to three different feature engineering processes (selection, transformation, and generation) which are executed iteratively, under the control of the user through the interface.
• Figure 3: Exploration of features with FeatureEnVi. The default slicing thresholds for the data space separate the instances into four quadrants that represent intervals of 25% predicted probability (see (a.1--a.4)). View (b) presents a table heatmap with five different feature selection techniques and their average value per feature. We exclude the less contributing features, as shown in the duplicated view (c). In the radial tree, the paths from (d.1) to (d.4) are the features for the groups formed at (a.1--a.4), respectively, while the features' impact for the entire data set is shown in the red box. The whole data space is displayed with even more details in the graph visualization in (e), where additional metrics' results are reported. A summary of the meaning of the visual encodings for these metrics is visible in the top-left corner in (e). More details about these views are described in the text.
• Figure 4: The feature transformation phase in order to empower features and improve the prediction. In (a), we have selected F1, and we check the impact of different transformation strategies in all slices of the data space. The entire data space is shown in more detail with the graph visualization in (b). The F1's feature transformation that appears the best is the logarithmic transformation (_l2 or _l10).
• Figure 5: The process of features' exploration in a vehicle recognition scenario. (a.1) to (a.4) depict the change of the thresholds for the different data slices to intensify the responses for borderline instances. In (b), the user excluded unimportant features and then validates the remaining features through the radial tree visualization. (c.1) to (c.4) contain the main statistical measures (target correlation and mutual information) and allow the user to discover that F8 and F16 should be excluded, as shown in (d). Finally, (e) illustrates the transformation alternatives of the most contributing feature (i.e., F4); the user employs a power function since it increases the correlation with the target and reduces slightly the correlation between features.