Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Decoupling Decision-Making in Fraud Prevention through Classifier Calibration for Business Logic Action

Emanuele Luzio, Moacir Antonelli Ponti, Christian Ramirez Arevalo, Luis Argerich

TL;DR

This paper tackles the problem of decoupling score-based actions from evolving ML models to preserve stable business logic. It proposes calibration-based decoupling and evaluates four post-processing methods—Platt Scaling, Isotonic Regression, Temperature Scaling, and Beta Calibration—on Bank Account Fraud data across three model families (CatBoost, LightGBM, and MLP). The study finds that isotonic and beta calibrations are particularly effective under training-testing drift, while decoupled systems generally maintain or improve performance and substantially improve maintainability and scalability. The results offer actionable guidance for practitioners seeking robust, modular fraud-detection pipelines that remain compliant with external requirements and resilient to distributional shifts.

Abstract

Machine learning models typically focus on specific targets like creating classifiers, often based on known population feature distributions in a business context. However, models calculating individual features adapt over time to improve precision, introducing the concept of decoupling: shifting from point evaluation to data distribution. We use calibration strategies as strategy for decoupling machine learning (ML) classifiers from score-based actions within business logic frameworks. To evaluate these strategies, we perform a comparative analysis using a real-world business scenario and multiple ML models. Our findings highlight the trade-offs and performance implications of the approach, offering valuable insights for practitioners seeking to optimize their decoupling efforts. In particular, the Isotonic and Beta calibration methods stand out for scenarios in which there is shift between training and testing data.

Decoupling Decision-Making in Fraud Prevention through Classifier Calibration for Business Logic Action

TL;DR

This paper tackles the problem of decoupling score-based actions from evolving ML models to preserve stable business logic. It proposes calibration-based decoupling and evaluates four post-processing methods—Platt Scaling, Isotonic Regression, Temperature Scaling, and Beta Calibration—on Bank Account Fraud data across three model families (CatBoost, LightGBM, and MLP). The study finds that isotonic and beta calibrations are particularly effective under training-testing drift, while decoupled systems generally maintain or improve performance and substantially improve maintainability and scalability. The results offer actionable guidance for practitioners seeking robust, modular fraud-detection pipelines that remain compliant with external requirements and resilient to distributional shifts.

Abstract

Machine learning models typically focus on specific targets like creating classifiers, often based on known population feature distributions in a business context. However, models calculating individual features adapt over time to improve precision, introducing the concept of decoupling: shifting from point evaluation to data distribution. We use calibration strategies as strategy for decoupling machine learning (ML) classifiers from score-based actions within business logic frameworks. To evaluate these strategies, we perform a comparative analysis using a real-world business scenario and multiple ML models. Our findings highlight the trade-offs and performance implications of the approach, offering valuable insights for practitioners seeking to optimize their decoupling efforts. In particular, the Isotonic and Beta calibration methods stand out for scenarios in which there is shift between training and testing data.
Paper Structure (17 sections, 7 equations, 3 figures, 2 tables)

This paper contains 17 sections, 7 equations, 3 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Objectives and scope
  3. Decoupling Thresholds from Evolving Models
  4. Calibration protocol for decoupled thresholds
  5. Bank Account Fraud dataset
  6. Calibration Techniques Overview
  7. Platt Scaling
  8. Isotonic Regression
  9. Temperature Scaling
  10. Beta Calibration
  11. Experimental Design
  12. Statistical test
  13. Results
  14. Conclusion
  15. Appendix
  16. ...and 2 more sections

Figures (3)

  • Figure 1: General approach for machine Learning model training and usage. The need
  • Figure 2: Decoupling business decision by calibration
  • Figure 3: Diagram for the Experimental Setup: obtaining bootstraps of the training set to train several models tuned and calibrated using a validation set, and evaluation using a test set.