Fighting Sampling Bias: A Framework for Training and Evaluating Credit Scoring Models

TL;DR

The paper tackles sampling bias in credit scoring by addressing bias in both training data (via bias-aware self-learning, BASL) and evaluation (via a Bayesian evaluation framework BM). It demonstrates, through synthetic simulations and a real high-dimensional micro-loan dataset with an unbiased holdout, that reject inference offers modest gains and can be fragile, whereas Bayesian evaluation provides more reliable performance estimates and supports better decision-making. BASL improves scorecard predictive performance by carefully selecting and labeling a subset of rejected cases to augment training without excessive bias amplification. In practice, Bayesian evaluation can yield substantial business value, including profit improvements, by guiding acceptance-rate decisions under bias. The work highlights practical data considerations, such as the need for unlabeled rejects data and privacy implications, and shows robustness across MAR and MNAR regimes with significant gains in high-dimensional settings.

Abstract

Scoring models support decision-making in financial institutions. Their estimation and evaluation are based on the data of previously accepted applicants with known repayment behavior. This creates sampling bias: the available labeled data offers a partial picture of the distribution of candidate borrowers, which the model is supposed to score. The paper addresses the adverse effect of sampling bias on model training and evaluation. To improve scorecard training, we propose bias-aware self-learning - a reject inference framework that augments the biased training data by inferring labels for selected rejected applications. For scorecard evaluation, we propose a Bayesian framework that extends standard accuracy measures to the biased setting and provides a reliable estimate of future scorecard performance. Extensive experiments on synthetic and real-world data confirm the superiority of our propositions over various benchmarks in predictive performance and profitability. By sensitivity analysis, we also identify boundary conditions affecting their performance. Notably, we leverage real-world data from a randomized controlled trial to assess the novel methodologies on holdout data that represent the true borrower population. Our findings confirm that reject inference is a difficult problem with modest potential to improve scorecard performance. Addressing sampling bias during scorecard evaluation is a much more promising route to improve scoring practices. For example, our results suggest a profit improvement of about eight percent, when using Bayesian evaluation to decide on acceptance rates.

Figures (6)

• Figure 1: Bias-Aware Self-Learning Framework
• Figure 2: Loss due to Sampling Bias and Gains from Our Propositions
• Figure 3: Sensitivity Analysis: Bias-Aware Self-Learning
• Figure 4: Sensitivity Analysis: Bayesian Evaluation
• Figure 5: Sensitivity Analysis: Missingness Type