The Effect of Balancing Methods on Model Behavior in Imbalanced Classification Problems

TL;DR

This work examines how data-balancing techniques for imbalanced classification impact model behavior, not just predictive performance. It combines Explainable AI tools—partial dependence profiles (PDP), accumulated local effects (ALE), and variable importance (VI)—with a new standard-deviation-of-differences (SDD) metric across logistic regression, random forest, and XGBoost on simulated and real datasets, implemented in the open-source edgaro package. A novel performance gain plot is proposed to jointly consider performance gains and behavioral changes, guiding the selection of balancing methods. Key findings show balancing can bias models toward balanced distributions and that behavior changes vary by method and model, motivating careful method choice and further improvements in imbalanced-learning evaluation.

Abstract

Imbalanced data poses a significant challenge in classification as model performance is affected by insufficient learning from minority classes. Balancing methods are often used to address this problem. However, such techniques can lead to problems such as overfitting or loss of information. This study addresses a more challenging aspect of balancing methods - their impact on model behavior. To capture these changes, Explainable Artificial Intelligence tools are used to compare models trained on datasets before and after balancing. In addition to the variable importance method, this study uses the partial dependence profile and accumulated local effects techniques. Real and simulated datasets are tested, and an open-source Python package edgaro is developed to facilitate this analysis. The results obtained show significant changes in model behavior due to balancing methods, which can lead to biased models toward a balanced distribution. These findings confirm that balancing analysis should go beyond model performance comparisons to achieve higher reliability of machine learning models. Therefore, we propose a new method performance gain plot for informed data balancing strategy to make an optimal selection of balancing method by analyzing the measure of change in model behavior versus performance gain.

Figures (10)

• Figure 1: Exemplary $PDP$ plots and the $SDD$ values.
• Figure 2: Balanced Accuracy of the models trained on the simulated dataset. group represents the $\beta_0$ values (group $i$: $\beta_{0i} = \{1.5, 2.5, 3.5, 4.5\}$), and var represents the variance of the error term ($var_j = \{1, 2, 3\}$)
• Figure 3: SDD results based on partial dependence profiles of the models trained on the simulated dataset. group represents the $\beta_0$ values (group $i$: $\beta_{0i} = \{1.5, 2.5, 3.5, 4.5\}$), and var represents the variance of the error term ($var_j = \{1, 2, 3\}$)
• Figure 4: SDD results based on accumulated local effect profiles of the models trained on the simulated dataset. group represents the $\beta_0$ values (group $i$: $\beta_{0i} = \{1.5, 2.5, 3.5, 4.5\}$), and var represents the variance of the error term ($var_j = \{1, 2, 3\}$)
• Figure 5: Balanced Accuracy results of the real dataset experiments.