Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Unraveling overoptimism and publication bias in ML-driven science

Pouria Saidi, Gautam Dasarathy, Visar Berisha

TL;DR

This paper addresses overoptimism in ML-driven science by modeling published accuracies with a parametric learning-curve framework $y(n)=A+\alpha n^{\beta}$ augmented by an overfitting term $\zeta n^{-0.5}$ and a publication-bias mechanism via truncation. It introduces a truncated-regression estimator to recover the true learning trajectory from censored data and solves the non-convex estimation problem with NSGA-II, providing bootstrapped confidence intervals. Through synthetic experiments and real-data meta-analyses in digital health (e.g., neuroimaging and speech-based brain-disorder classification), the method demonstrates the ability to recover realistic performance limits and quantify the extent of overoptimism across fields. The approach offers a principled way to debias reported ML results, guiding more trustworthy interpretations and deployments, especially when large-scale data are unavailable.

Abstract

Machine Learning (ML) is increasingly used across many disciplines with impressive reported results. However, recent studies suggest published performance of ML models are often overoptimistic. Validity concerns are underscored by findings of an inverse relationship between sample size and reported accuracy in published ML models, contrasting with the theory of learning curves where accuracy should improve or remain stable with increasing sample size. This paper investigates factors contributing to overoptimism in ML-driven science, focusing on overfitting and publication bias. We introduce a novel stochastic model for observed accuracy, integrating parametric learning curves and the aforementioned biases. We construct an estimator that corrects for these biases in observed data. Theoretical and empirical results show that our framework can estimate the underlying learning curve, providing realistic performance assessments from published results. Applying the model to meta-analyses of classifications of neurological conditions, we estimate the inherent limits of ML-based prediction in each domain.

Unraveling overoptimism and publication bias in ML-driven science

TL;DR

This paper addresses overoptimism in ML-driven science by modeling published accuracies with a parametric learning-curve framework augmented by an overfitting term and a publication-bias mechanism via truncation. It introduces a truncated-regression estimator to recover the true learning trajectory from censored data and solves the non-convex estimation problem with NSGA-II, providing bootstrapped confidence intervals. Through synthetic experiments and real-data meta-analyses in digital health (e.g., neuroimaging and speech-based brain-disorder classification), the method demonstrates the ability to recover realistic performance limits and quantify the extent of overoptimism across fields. The approach offers a principled way to debias reported ML results, guiding more trustworthy interpretations and deployments, especially when large-scale data are unavailable.

Abstract

Machine Learning (ML) is increasingly used across many disciplines with impressive reported results. However, recent studies suggest published performance of ML models are often overoptimistic. Validity concerns are underscored by findings of an inverse relationship between sample size and reported accuracy in published ML models, contrasting with the theory of learning curves where accuracy should improve or remain stable with increasing sample size. This paper investigates factors contributing to overoptimism in ML-driven science, focusing on overfitting and publication bias. We introduce a novel stochastic model for observed accuracy, integrating parametric learning curves and the aforementioned biases. We construct an estimator that corrects for these biases in observed data. Theoretical and empirical results show that our framework can estimate the underlying learning curve, providing realistic performance assessments from published results. Applying the model to meta-analyses of classifications of neurological conditions, we estimate the inherent limits of ML-based prediction in each domain.
Paper Structure (12 sections, 2 theorems, 15 equations, 7 figures, 6 tables)

This paper contains 12 sections, 2 theorems, 15 equations, 7 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Proposed model
  3. Contributions
  4. Results
  5. Methods
  6. Observation model for reported classification accuracy
  7. Proposed solution
  8. Limitations
  9. Conclusion
  10. On identifiability of the parameters of a truncated Gaussian distribution
  11. Data generating process for Experiment 2
  12. Extended results

Key Result

Theorem 1

Let $\mathrm{N}$ denote the set of all sample sizes used to train ML models in a field. For every $n \in \mathrm{N}$, assume access to $M$ samples (reported accuracies) that are independently drawn from a univariate Gaussian distribution with mean $\mu_n$ and variance $\sigma_n^2$, and have survived

Figures (7)

  • Figure 1: The reported accuracy vs. sample size from a collection of results in a meta-analysis study arbabshirani2017single. This analysis considers neuroimaging-based classification models between a control group and a patient cohort with (a) Alzheimer's disease (AD), and (b) Schizophrenia. (c) An empirical learning curve that is obtained by solving a binary classification problem using different ML models and sample sizes. The y-axis is in linear scale and the x-axis is in log scale.
  • Figure 2: Evaluation of the proposed method per Experiment 1 where we sample from the observation model for (a) Problem 1: $A = 0.78$, $\alpha = -1.24$, $\beta = -0.76$, $\zeta = 0.45$ and $c_1 = 0.50$. (b) Problem 2: $A = 0.75$, $\alpha = -0.75$, $\beta = -0.57$, $\zeta = 0.85$ and $c_1 = 0.40$. The results show the overoptimistic accuracies (blue circles), fit to the overoptimistic results (blue line), the new estimates of the learning curve (red line) along the true learning curve (green line). The y-axis is in linear and x-axis is in log scale.
  • Figure 3: Evaluation of the proposed method per Experiment 2 where we simulate overfitting and publication bias in binary classification, (a) Problem 1, (b) Problem 2. The results show the overoptimistic accuracies (blue circles), fit to the overoptimistic results (blue line), the new estimates of the learning curve (red line) along the true learning curve (green line). The y-axis is in linear and the x-axis is in log scale.
  • Figure 4: Estimates of the learning trajectories of ML models from (a) meta-analyses of neuroimaging-based prediction of AD arbabshirani2017single and (b) speech-based prediction of AD de2020artificialpetti2020systematicmartinez2021ten per Experiment 3. The results show the new estimates of the empirical learning curve (red), with the published results (blue circles) and fit to the observations (blue). Faded blue circles shown are considered as outliers and were removed from analysis. y-axis is in linear and x-axis is in log scale.
  • Figure 5: Extended results based on Experiment 1 and for 5 additional cases introduced in Section \ref{['subsec:extended']}.
  • ...and 2 more figures

Theorems & Definitions (2)

  • Theorem 1
  • Lemma 2: Special case of Theorem 1 in kontonis2019efficient