How Ensemble Learning Balances Accuracy and Overfitting: A Bias-Variance Perspective on Tabular Data

TL;DR

Ensemble methods balance accuracy and overfitting on tabular data by navigating bias-variance trade-offs across regimes. Using five-seed repeated stratified cross-validation and paired significance testing, the study contrasts linear models, single trees, and nine ensembles across four datasets, and introduces simple dataset-complexity indicators to predict when ensembles help. It identifies three regimes—clean/near-linear, structured nonlinear, and noisy/imbalanced—and provides practical guidelines for model selection that balance predictive gains against variance, cost, and minority-class performance. The findings reaffirm when tree-based ensembles are advantageous and emphasize regularization and appropriate evaluation for high-stakes, real-world tasks.

Abstract

Ensemble models often achieve higher accuracy than single learners, but their ability to maintain small generalization gaps is not always well understood. This study examines how ensembles balance accuracy and overfitting across four tabular classification tasks: Breast Cancer, Heart Disease, Pima Diabetes, and Credit Card Fraud. Using repeated stratified cross validation with statistical significance testing, we compare linear models, a single decision tree, and nine ensemble methods. The results show that ensembles can reach high accuracy without large gaps by reducing variance through averaging or controlled boosting. On nearly linear and clean data, linear models already generalize well and ensembles offer little additional benefit. On datasets with meaningful nonlinear structure, tree based ensembles increase test accuracy by 5 to 7 points while keeping gaps below 3 percent. On noisy or highly imbalanced datasets, ensembles remain competitive but require regularization to avoid fitting noise or majority class patterns. We also compute simple dataset complexity indicators, such as linearity score, Fisher ratio, and noise estimate, which explain when ensembles are likely to control variance effectively. Overall, the study provides a clear view of how and when ensembles maintain high accuracy while keeping overfitting low, offering practical guidance for model selection in real world tabular applications.

Figures (9)

• Figure 1: Cross-validated train and test accuracy (mean $\pm$ std across 5 seeds) on Breast Cancer. Linear models achieve high and closely aligned train/test accuracy; ensembles reach near-perfect train accuracy with only modest test gains.
• Figure 2: Accuracy-based generalization gap for Breast Cancer. Ensembles show slightly larger gaps than linear models, but gaps remain small overall on this nearly linearly separable task.
• Figure 3: Cross-validated train and test accuracy on Heart Disease. Tree ensembles substantially improve test accuracy over linear baselines and a single Decision Tree, with statistical significance $p < 0.01$.
• Figure 4: Generalization gap for Heart Disease. Ensembles maintain small gaps despite higher capacity, indicating effective variance reduction that is well-matched to the data structure.
• Figure 5: Generalization gap for Pima Indians Diabetes. Some high-capacity ensembles nearly memorize training data without achieving strong test gains, leading to larger gaps, while better-regularized models show more moderate gaps.