Data Leakage and Deceptive Performance: A Critical Examination of Credit Card Fraud Detection Methodologies

TL;DR

The paper addresses methodological flaws in credit card fraud detection research, arguing that data leakage and vague reporting can inflate results and misrepresent model capability. It demonstrates this through a deliberately flawed evaluation: applying SMOTE before train/test split with a simple MLP yields deceptively high metrics, illustrating that evaluation protocol can overshadow algorithmic sophistication. The authors identify four persistent issues—data leakage, vagueness in methods, inadequate temporal validation, and recall-focused metric manipulation—and advocate for stricter, transparent evaluation practices. The work underscores that rigorous methodology must precede architectural complexity to ensure findings generalize to real-world fraud detection tasks and informs better research practices across ML applications.

Abstract

This study critically examines the methodological rigor in credit card fraud detection research, revealing how fundamental evaluation flaws can overshadow algorithmic sophistication. Through deliberate experimentation with improper evaluation protocols, we demonstrate that even simple models can achieve deceptively impressive results when basic methodological principles are violated. Our analysis identifies four critical issues plaguing current approaches: (1) pervasive data leakage from improper preprocessing sequences, (2) intentional vagueness in methodological reporting, (3) inadequate temporal validation for transaction data, and (4) metric manipulation through recall optimization at precision's expense. We present a case study showing how a minimal neural network architecture with data leakage outperforms many sophisticated methods reported in literature, achieving 99.9\% recall despite fundamental evaluation flaws. These findings underscore that proper evaluation methodology matters more than model complexity in fraud detection research. The study serves as a cautionary example of how methodological rigor must precede architectural sophistication, with implications for improving research practices across machine learning applications.

Figures (5)

• Figure 1: Snapshot of the first five rows of the dataset.
• Figure 2: A generic MLP architecture
• Figure 3: The Flawed MLP Model
• Figure 4: Test results after applying the MLP with no hidden layer ($N=0$).
• Figure 5: Pairwise PRC(left) and ROC (right) test results of the MLP method with $N$ neurons in a single hidden layer.