Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

A Reproducible Framework for Bias-Resistant Machine Learning on Small-Sample Neuroimaging Data

Jagan Mohan Reddy Dwarampudi, Jennifer L Purks, Joshua Wong, Renjie Hu, Tania Banerjee

TL;DR

The paper tackles optimistic bias in small-sample neuroimaging by proposing a reproducible framework that fuses domain-informed feature engineering, strictly nested cross-validation, and calibrated decision-threshold optimization. It demonstrates that, on $n=332$ structural MRI cases predicting DBS cognitive outcomes, nested CV yields a balanced accuracy of about $0.66 \pm 0.06$ and an AUC-ROC of about $0.72 \pm 0.06$, with robust, threshold-stable operating points around $t \approx 0.39$. The combination of interpretable morphometric composites and unbiased evaluation provides deployment-ready, probabilistically calibrated predictions, mitigating leakage and overfitting in high-dimensional, low-sample biomedical settings. While limited by single-site data, the framework is model- and modality-agnostic and offers a principled blueprint for reproducible machine learning in data-constrained neuroimaging contexts with potential clinical utility for DBS candidacy assessment and beyond.

Abstract

We introduce a reproducible, bias-resistant machine learning framework that integrates domain-informed feature engineering, nested cross-validation, and calibrated decision-threshold optimization for small-sample neuroimaging data. Conventional cross-validation frameworks that reuse the same folds for both model selection and performance estimation yield optimistically biased results, limiting reproducibility and generalization. Demonstrated on a high-dimensional structural MRI dataset of deep brain stimulation cognitive outcomes, the framework achieved a nested-CV balanced accuracy of 0.660\,$\pm$\,0.068 using a compact, interpretable subset selected via importance-guided ranking. By combining interpretability and unbiased evaluation, this work provides a generalizable computational blueprint for reliable machine learning in data-limited biomedical domains.

A Reproducible Framework for Bias-Resistant Machine Learning on Small-Sample Neuroimaging Data

TL;DR

The paper tackles optimistic bias in small-sample neuroimaging by proposing a reproducible framework that fuses domain-informed feature engineering, strictly nested cross-validation, and calibrated decision-threshold optimization. It demonstrates that, on structural MRI cases predicting DBS cognitive outcomes, nested CV yields a balanced accuracy of about and an AUC-ROC of about , with robust, threshold-stable operating points around . The combination of interpretable morphometric composites and unbiased evaluation provides deployment-ready, probabilistically calibrated predictions, mitigating leakage and overfitting in high-dimensional, low-sample biomedical settings. While limited by single-site data, the framework is model- and modality-agnostic and offers a principled blueprint for reproducible machine learning in data-constrained neuroimaging contexts with potential clinical utility for DBS candidacy assessment and beyond.

Abstract

We introduce a reproducible, bias-resistant machine learning framework that integrates domain-informed feature engineering, nested cross-validation, and calibrated decision-threshold optimization for small-sample neuroimaging data. Conventional cross-validation frameworks that reuse the same folds for both model selection and performance estimation yield optimistically biased results, limiting reproducibility and generalization. Demonstrated on a high-dimensional structural MRI dataset of deep brain stimulation cognitive outcomes, the framework achieved a nested-CV balanced accuracy of 0.660\,\,0.068 using a compact, interpretable subset selected via importance-guided ranking. By combining interpretability and unbiased evaluation, this work provides a generalizable computational blueprint for reliable machine learning in data-limited biomedical domains.
Paper Structure (9 sections, 1 figure, 5 tables)

This paper contains 9 sections, 1 figure, 5 tables.

Table of Contents

  1. Introduction
  2. Methodology
  3. Results
  4. Nested cross-validated performance
  5. Feature engineering and threshold stability
  6. Calibration and bias control
  7. Conclusion
  8. Compliance with Ethical Standards
  9. Conflicts of Interest

Figures (1)

  • Figure 1: Receiver-operating characteristic (ROC) curve and calibration reliability diagram for the Random Forest trained on feature-engineered SynthSeg features under nested CV with sigmoid calibration (Platt scaling). Mean AUC--ROC $\approx$ 0.72, Brier $\approx$ 0.22, and ECE $\approx$ 0.13. The mean AUC–ROC $\approx$ 0.72 reflects ranking performance across thresholds, whereas the balanced accuracy in Table \ref{['tab:synthseg-ba-calibrated']} reports accuracy at the optimized decision threshold.