Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

An Experimental Study on the Rashomon Effect of Balancing Methods in Imbalanced Classification

Mustafa Cavus, Przemysław Biecek

TL;DR

The paper addresses the risk of biased predictions in imbalanced classification due to the Rashomon effect, where many near-equally accurate models exist. It proposes a data-centric framework that uses obscurity and established Rashomon-set metrics to quantify predictive multiplicity, constructing empirical Rashomon sets with an AutoML workflow under balancing schemes. Key findings show that balancing methods inflate multiplicity while partial resampling fails to mitigate it, and an extended performance-gain plot provides a practical tool to balance accuracy with multiplicity. The work offers actionable guidance for responsible data-centric AI, highlighting multiplicity risks and proposing monitoring tools to guide balancing strategy choices in imbalanced domains.

Abstract

Predictive models may generate biased predictions when classifying imbalanced datasets. This happens when the model favors the majority class, leading to low performance in accurately predicting the minority class. To address this issue, balancing or resampling methods are critical data-centric AI approaches in the modeling process to improve prediction performance. However, there have been debates and questions about the functionality of these methods in recent years. In particular, many candidate models may exhibit very similar predictive performance, called the Rashomon effect, in model selection, and they may even produce different predictions for the same observations. Selecting one of these models without considering the predictive multiplicity -- which is the case of yielding conflicting models' predictions for any sample -- can result in blind selection. In this paper, the impact of balancing methods on predictive multiplicity is examined using the Rashomon effect. It is crucial because the blind model selection in data-centric AI is risky from a set of approximately equally accurate models. This may lead to severe problems in model selection, validation, and explanation. To tackle this matter, we conducted real dataset experiments to observe the impact of balancing methods on predictive multiplicity through the Rashomon effect by using a newly proposed metric obscurity in addition to the existing ones: ambiguity and discrepancy. Our findings showed that balancing methods inflate the predictive multiplicity and yield varying results. To monitor the trade-off between the prediction performance and predictive multiplicity for conducting the modeling process responsibly, we proposed using the extended version of the performance-gain plot when balancing the training data.

An Experimental Study on the Rashomon Effect of Balancing Methods in Imbalanced Classification

TL;DR

The paper addresses the risk of biased predictions in imbalanced classification due to the Rashomon effect, where many near-equally accurate models exist. It proposes a data-centric framework that uses obscurity and established Rashomon-set metrics to quantify predictive multiplicity, constructing empirical Rashomon sets with an AutoML workflow under balancing schemes. Key findings show that balancing methods inflate multiplicity while partial resampling fails to mitigate it, and an extended performance-gain plot provides a practical tool to balance accuracy with multiplicity. The work offers actionable guidance for responsible data-centric AI, highlighting multiplicity risks and proposing monitoring tools to guide balancing strategy choices in imbalanced domains.

Abstract

Predictive models may generate biased predictions when classifying imbalanced datasets. This happens when the model favors the majority class, leading to low performance in accurately predicting the minority class. To address this issue, balancing or resampling methods are critical data-centric AI approaches in the modeling process to improve prediction performance. However, there have been debates and questions about the functionality of these methods in recent years. In particular, many candidate models may exhibit very similar predictive performance, called the Rashomon effect, in model selection, and they may even produce different predictions for the same observations. Selecting one of these models without considering the predictive multiplicity -- which is the case of yielding conflicting models' predictions for any sample -- can result in blind selection. In this paper, the impact of balancing methods on predictive multiplicity is examined using the Rashomon effect. It is crucial because the blind model selection in data-centric AI is risky from a set of approximately equally accurate models. This may lead to severe problems in model selection, validation, and explanation. To tackle this matter, we conducted real dataset experiments to observe the impact of balancing methods on predictive multiplicity through the Rashomon effect by using a newly proposed metric obscurity in addition to the existing ones: ambiguity and discrepancy. Our findings showed that balancing methods inflate the predictive multiplicity and yield varying results. To monitor the trade-off between the prediction performance and predictive multiplicity for conducting the modeling process responsibly, we proposed using the extended version of the performance-gain plot when balancing the training data.
Paper Structure (5 sections, 7 equations, 6 figures, 1 table)

This paper contains 5 sections, 7 equations, 6 figures, 1 table.

Table of Contents

  1. Introduction
  2. Preliminaries
  3. Experiments
  4. Results
  5. Conclusions

Figures (6)

  • Figure 1: The illustration of relations among ambiguity, discrepancy, and obscurity.
  • Figure 2: The 2d density plot of the Rashomon metrics obscurity and discrepancy for different balancing methods and various resampling ratios.
  • Figure 3: The distribution plots of the Rashomon metrics obscurity and discrepancy for different balancing methods.
  • Figure 4: The distribution plots of the Rashomon metric variable importance order discrepancy for different balancing methods.
  • Figure 5: The distribution plots of the Rashomon metric variable importance order discrepancy for different balancing methods and varying partial resampling ratios.
  • ...and 1 more figures