Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Best Practices for Machine Learning Experimentation in Scientific Applications

Umberto Michelucci, Francesca Venturini

TL;DR

The paper addresses reliability gaps in ML experiments for scientific problems by proposing a practical, reproducible workflow from raw data to evaluation. It introduces metrics for overfitting and instability, notably the Logarithmic Overfitting Ratio (LOR) and Composite Overfitting Score (COS), to complement traditional validation. Emphasizing simple baselines, consistent preprocessing, and transparent reporting, it covers dataset versioning, cross-validation strategies, and clear result presentation to enable fair comparisons between classical and deep learning approaches. Together, these guidelines help researchers establish robust baselines and draw credible, evidence-based conclusions from ML-driven scientific analyses.

Abstract

Machine learning (ML) is increasingly adopted in scientific research, yet the quality and reliability of results often depend on how experiments are designed and documented. Poor baselines, inconsistent preprocessing, or insufficient validation can lead to misleading conclusions about model performance. This paper presents a practical and structured guide for conducting ML experiments in scientific applications, focussing on reproducibility, fair comparison, and transparent reporting. We outline a step-by-step workflow, from dataset preparation to model selection and evaluation, and propose metrics that account for overfitting and instability across validation folds, including the Logarithmic Overfitting Ratio (LOR) and the Composite Overfitting Score (COS). Through recommended practices and example reporting formats, this work aims to support researchers in establishing robust baselines and drawing valid evidence-based insights from ML models applied to scientific problems.

Best Practices for Machine Learning Experimentation in Scientific Applications

TL;DR

The paper addresses reliability gaps in ML experiments for scientific problems by proposing a practical, reproducible workflow from raw data to evaluation. It introduces metrics for overfitting and instability, notably the Logarithmic Overfitting Ratio (LOR) and Composite Overfitting Score (COS), to complement traditional validation. Emphasizing simple baselines, consistent preprocessing, and transparent reporting, it covers dataset versioning, cross-validation strategies, and clear result presentation to enable fair comparisons between classical and deep learning approaches. Together, these guidelines help researchers establish robust baselines and draw credible, evidence-based conclusions from ML-driven scientific analyses.

Abstract

Machine learning (ML) is increasingly adopted in scientific research, yet the quality and reliability of results often depend on how experiments are designed and documented. Poor baselines, inconsistent preprocessing, or insufficient validation can lead to misleading conclusions about model performance. This paper presents a practical and structured guide for conducting ML experiments in scientific applications, focussing on reproducibility, fair comparison, and transparent reporting. We outline a step-by-step workflow, from dataset preparation to model selection and evaluation, and propose metrics that account for overfitting and instability across validation folds, including the Logarithmic Overfitting Ratio (LOR) and the Composite Overfitting Score (COS). Through recommended practices and example reporting formats, this work aims to support researchers in establishing robust baselines and drawing valid evidence-based insights from ML models applied to scientific problems.

Paper Structure

This paper contains 10 sections, 4 equations, 1 figure, 4 tables.

Table of Contents

  1. Introduction
  2. Design of Experiments
  3. Dataset Preparation
  4. Choosing classical models for the experiments
  5. Cross-Validation (CV) and Evaluation
  6. Logarithimic Overfitting Ratio
  7. Extending to Deep Learning
  8. Selecting Meaningful Results
  9. Result Table and Reporting
  10. Aknowledgments

Figures (1)

  • Figure 1: An example on how a table summarising results of experiments might look like. Note that this table reports only metrics and no other information, since in the project we generated one table for each experiment. But it should give you an idea about how it could look like. The red lines are those for which $R^2<0$, yellow those for which $R^2<0.85$ and green ones for which $R^2>0.85$. Model instnaces that have $R^2>0.85$ are the one that should be considered. In simple bold face we have highlighted the model instance for the LOR closest to zero, while in bold face and italics the one with the COS closest to 1.