The Impact of Feature Scaling In Machine Learning: Effects on Regression and Classification Tasks

TL;DR

The paper tackles the lack of comprehensive benchmarks for feature scaling by systematically evaluating 12 scaling techniques across 14 algorithms on 16 tabular datasets for classification and regression. It leverages a rigorous, reproducible workflow with 70/30 train/test splits and open-source code, revealing that ensemble methods are largely robust to scaling while linear and distance-based models exhibit strong sensitivity to the scaler choice. Statistical tests (Wilcoxon signed-rank and Friedman) quantify when scaling impacts performance, indicating concrete guidance on model-scaler pairings and the cost-benefit trade-offs in computation and memory. The work emphasizes reproducibility and provides actionable insights for practitioners, such as prioritizing ensemble models to avoid unnecessary preprocessing, and highlights memory and time considerations tied to certain scalers. Overall, this study offers a comprehensive, transparent benchmark that informs principled feature-scaling decisions in real-world tabular-data modeling.

Abstract

This research addresses the critical lack of comprehensive studies on feature scaling by systematically evaluating 12 scaling techniques - including several less common transformations - across 14 different Machine Learning algorithms and 16 datasets for classification and regression tasks. We meticulously analyzed impacts on predictive performance (using metrics such as accuracy, MAE, MSE, and $R^2$) and computational costs (training time, inference time, and memory usage). Key findings reveal that while ensemble methods (such as Random Forest and gradient boosting models like XGBoost, CatBoost and LightGBM) demonstrate robust performance largely independent of scaling, other widely used models such as Logistic Regression, SVMs, TabNet, and MLPs show significant performance variations highly dependent on the chosen scaler. This extensive empirical analysis, with all source code, experimental results, and model parameters made publicly available to ensure complete transparency and reproducibility, offers model-specific crucial guidance to practitioners on the need for an optimal selection of feature scaling techniques.

Figures (10)

• Figure 1: Experimental design.
• Figure 2: Accuracy results for the 9 datasets. The ensemble models are: RF, LGBM, CatBoost, and XGBoost.
• Figure 3: MSE results for the 7 datasets. The ensemble models are: RF, LGBM, CatBoost, and XGBoost.
• Figure 4: $R^2$ results for the 7 datasets. The ensemble models are: RF, LGBM, CatBoost, and XGBoost.
• Figure 5: MAE results for the 7 datasets. The ensemble models are: RF, LGBM, CatBoost, and XGBoost.