Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Squeezing Lemons with Hammers: An Evaluation of AutoML and Tabular Deep Learning for Data-Scarce Classification Applications

Ricardo Knauer, Erik Rodner

TL;DR

This work tackles the problem of selecting ML approaches for tabular data when samples are scarce. It compares AutoML frameworks (AutoPrognosis, AutoGluon) and pretrained tabular DL models (TabPFN, HyperFast) against a simple L2-regularized logistic regression baseline on 44 very-small binary datasets from the PMLB collection, with median size $204$ and range $[32, 500]$ samples. The study finds that logistic regression achieves comparable discriminative performance to the more complex methods on about $55\%$ of datasets (within $3\%$ of the best approach in many cases), and sometimes outperforms them due to overfitting in the latter. A meta-feature analysis shows that simple dataset features provide only weak guidance for method choice, reinforcing a practical baseline-first recommendation and offering the per-dataset best L2 hyperparameters to support meta-learning in data-scarce settings. Overall, the results argue for starting with a transparent logistic regression baseline in data-scarce tabular applications and resorting to AutoML or pretrained DL methods only when additional performance is required, with an emphasis on interpretability when possible.

Abstract

Many industry verticals are confronted with small-sized tabular data. In this low-data regime, it is currently unclear whether the best performance can be expected from simple baselines, or more complex machine learning approaches that leverage meta-learning and ensembling. On 44 tabular classification datasets with sample sizes $\leq$ 500, we find that L2-regularized logistic regression performs similar to state-of-the-art automated machine learning (AutoML) frameworks (AutoPrognosis, AutoGluon) and off-the-shelf deep neural networks (TabPFN, HyperFast) on the majority of the benchmark datasets. We therefore recommend to consider logistic regression as the first choice for data-scarce applications with tabular data and provide practitioners with best practices for further method selection.

Squeezing Lemons with Hammers: An Evaluation of AutoML and Tabular Deep Learning for Data-Scarce Classification Applications

TL;DR

This work tackles the problem of selecting ML approaches for tabular data when samples are scarce. It compares AutoML frameworks (AutoPrognosis, AutoGluon) and pretrained tabular DL models (TabPFN, HyperFast) against a simple L2-regularized logistic regression baseline on 44 very-small binary datasets from the PMLB collection, with median size and range samples. The study finds that logistic regression achieves comparable discriminative performance to the more complex methods on about of datasets (within of the best approach in many cases), and sometimes outperforms them due to overfitting in the latter. A meta-feature analysis shows that simple dataset features provide only weak guidance for method choice, reinforcing a practical baseline-first recommendation and offering the per-dataset best L2 hyperparameters to support meta-learning in data-scarce settings. Overall, the results argue for starting with a transparent logistic regression baseline in data-scarce tabular applications and resorting to AutoML or pretrained DL methods only when additional performance is required, with an emphasis on interpretability when possible.

Abstract

Many industry verticals are confronted with small-sized tabular data. In this low-data regime, it is currently unclear whether the best performance can be expected from simple baselines, or more complex machine learning approaches that leverage meta-learning and ensembling. On 44 tabular classification datasets with sample sizes 500, we find that L2-regularized logistic regression performs similar to state-of-the-art automated machine learning (AutoML) frameworks (AutoPrognosis, AutoGluon) and off-the-shelf deep neural networks (TabPFN, HyperFast) on the majority of the benchmark datasets. We therefore recommend to consider logistic regression as the first choice for data-scarce applications with tabular data and provide practitioners with best practices for further method selection.
Paper Structure (8 sections, 3 figures)

This paper contains 8 sections, 3 figures.

Table of Contents

  1. Introduction
  2. Related work
  3. Experiments
  4. Experimental setup
  5. Experimental results
  6. Meta-feature analysis
  7. Conclusion
  8. Appendix

Figures (3)

  • Figure 1: Mean test AUCs with their medians and interquartile ranges shown across sample size ranges in steps of 100 for AutoML, deep learning, and logistic regression.
  • Figure 2: Critical difference diagram to detect pairwise mean test AUC differences, based on the Holm-adjusted Wilcoxon signed-rank test demsar2006statisticalbenavoli2016should, for AutoML, deep learning, and logistic regression. Approaches that are not statistically different at the 0.05 significance level are connected by a bold vertical bar. $^\mathbf{\star}$Note that TabPFN results are biased (Sect. \ref{['sec:results']}).
  • Figure 3: Top-3 meta-features according to their relationship with the mean test AUC differences between each AutoML / deep learning method and logistic regression, based on absolute Spearman rank correlations. Positive correlation coefficients are shown in red, negative coefficients in blue.