Investigating the Impact of Balancing, Filtering, and Complexity on Predictive Multiplicity: A Data-Centric Perspective

TL;DR

This paper investigates how data preprocessing techniques like balancing and filtering methods impact predictive multiplicity and model stability, considering the complexity of the data, and assess the level of predictive multiplicity introduced by these methods by leveraging the Rashomon effect.

Abstract

The Rashomon effect presents a significant challenge in model selection. It occurs when multiple models achieve similar performance on a dataset but produce different predictions, resulting in predictive multiplicity. This is especially problematic in high-stakes environments, where arbitrary model outcomes can have serious consequences. Traditional model selection methods prioritize accuracy and fail to address this issue. Factors such as class imbalance and irrelevant variables further complicate the situation, making it harder for models to provide trustworthy predictions. Data-centric AI approaches can mitigate these problems by prioritizing data optimization, particularly through preprocessing techniques. However, recent studies suggest preprocessing methods may inadvertently inflate predictive multiplicity. This paper investigates how data preprocessing techniques like balancing and filtering methods impact predictive multiplicity and model stability, considering the complexity of the data. We conduct the experiments on 21 real-world datasets, applying various balancing and filtering techniques, and assess the level of predictive multiplicity introduced by these methods by leveraging the Rashomon effect. Additionally, we examine how filtering techniques reduce redundancy and enhance model generalization. The findings provide insights into the relationship between balancing methods, data complexity, and predictive multiplicity, demonstrating how data-centric AI strategies can improve model performance.

Figures (7)

• Figure 1: Illustration of a Rashomon cube with size 5 means that comprises five models and observations. The discrepancy is the maximum conflict ratio between the models, and obscurity shows the mean conflict ratio across the observations.
• Figure 2: The similarity analysis of datasets from our benchmark. Similarity is presented with the cluster plot and the PCA plot. Similarity is calculated based on the data complexity metrics
• Figure 3: The distribution of the disagreement metrics obscurity and discrepancy for the balancing methods.
• Figure 4: The 2d density plot of the disagreement metrics obscurity and discrepancy for the balancing and filtering methods.
• Figure 5: The 2d density plot of the disagreement metrics obscurity and discrepancy for the filtering methods and the complexity of the dataset.