A Data-Centric Perspective on Evaluating Machine Learning Models for Tabular Data

TL;DR

This work challenges the prevailing model-centric evaluation paradigm for tabular data by introducing a data-centric framework that pairs real-world Kaggle datasets with dataset-specific expert preprocessing, including feature engineering and test-time adaptation. By separating preprocessing from modeling and using external leaderboard references, the study shows that model rankings are highly sensitive to dataset-specific FE and TTA, with FE remaining the primary driver of top performance even for recent architectures. The findings argue for benchmarks that reflect practitioners' pipelines and distribution-shift realities, and they outline directions for future work in data-aware evaluation, temporal adaptations, and benchmarks aligned with real-world needs. Overall, the framework provides a realistic, extensible platform for evaluating and advancing tabular-data methods beyond AutoML-centric baselines.

Abstract

Tabular data is prevalent in real-world machine learning applications, and new models for supervised learning of tabular data are frequently proposed. Comparative studies assessing the performance of models typically consist of model-centric evaluation setups with overly standardized data preprocessing. This paper demonstrates that such model-centric evaluations are biased, as real-world modeling pipelines often require dataset-specific preprocessing and feature engineering. Therefore, we propose a data-centric evaluation framework. We select 10 relevant datasets from Kaggle competitions and implement expert-level preprocessing pipelines for each dataset. We conduct experiments with different preprocessing pipelines and hyperparameter optimization (HPO) regimes to quantify the impact of model selection, HPO, feature engineering, and test-time adaptation. Our main findings are: 1. After dataset-specific feature engineering, model rankings change considerably, performance differences decrease, and the importance of model selection reduces. 2. Recent models, despite their measurable progress, still significantly benefit from manual feature engineering. This holds true for both tree-based models and neural networks. 3. While tabular data is typically considered static, samples are often collected over time, and adapting to distribution shifts can be important even in supposedly static data. These insights suggest that research efforts should be directed toward a data-centric perspective, acknowledging that tabular data requires feature engineering and often exhibits temporal characteristics. Our framework is available under: https://github.com/atschalz/dc_tabeval.

Figures (12)

• Figure 1: Illustration of the components of our evaluation framework.
• Figure 2: Illustration of the dataset selection process. Details on the criteria and all screened datasets can be found in the Appendix. The Figure only lists the competitions as temporal, which were not already excluded for other reasons. In total, we identified 46 competition datasets with temporal characteristics (i.e., timestamps as a feature). Consistent with related work, we include competitions that have timestamps but can be approached without time-sensitive feature engineering.
• Figure 3: Performance gains from different modeling components on the private Kaggle leaderboard by dataset and model. Higher values correspond to a better position. Each segment represents the performance gain of adding the modeling component to the previous configuration. 'Default' corresponds to the model performance with default hyperparameters in a standardized preprocessing pipeline. Light and extensive HPO correspond to tuning hyperparameters in the same preprocessing pipeline. Expert FE and FE-TTA correspond to the model performance with extensively tuned hyperparameters in the feature engineering and the test-time adaptation pipeline respectively.
• Figure 4: Average leaderboard position of models with different preprocessing. Black horizontal lines denote the Spearman correlation between all results with the respective preprocessing.
• Figure 5: Progress made through recent models trained in the standardized preprocessing pipeline, illustrated by retrospective comparison to the Kaggle leaderboard. Best NN denotes the best model of FTTransformer, MLP-PLR, and GRANDE.