Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Verification and Validation for Trustworthy Scientific Machine Learning

John D. Jakeman, Lorena A. Barba, Joaquim R. R. A. Martins, Thomas O'Leary-Roseberry

TL;DR

The paper addresses the trustworthiness of Scientific Machine Learning (SciML) by outlining consensus-based verification, validation, and continuous credibility practices that adapt established CSE standards to SciML's data-driven context. It proposes a four-component framework—problem definition, verification, validation, and continuous credibility building—with 16 concrete recommendations to guide rigorous reporting, data provenance, and reproducibility. Through qualitative discussion and real-world examples, the work highlights challenges such as data dependence, nonconvex optimization, and uncertain theory, while offering practical strategies for robust calibration, uncertainty quantification, and cross-domain evaluation. The contributions aim to enable trustworthy SciML across high-consequence domains by fostering transparency, standardization, and community-wide adoption of V&V practices that bridge CSE rigor with SciML innovation.

Abstract

Scientific machine learning (SciML) models are transforming many scientific disciplines. However, the development of good modeling practices to increase the trustworthiness of SciML has lagged behind its application, limiting its potential impact. The goal of this paper is to start a discussion on establishing consensus-based good practices for predictive SciML. We identify key challenges in applying existing computational science and engineering guidelines, such as verification and validation protocols, and provide recommendations to address these challenges. Our discussion focuses on predictive SciML, which uses machine learning models to learn, improve, and accelerate numerical simulations of physical systems. While centered on predictive applications, our 16 recommendations aim to help researchers conduct and document their modeling processes rigorously across all SciML domains.

Verification and Validation for Trustworthy Scientific Machine Learning

TL;DR

The paper addresses the trustworthiness of Scientific Machine Learning (SciML) by outlining consensus-based verification, validation, and continuous credibility practices that adapt established CSE standards to SciML's data-driven context. It proposes a four-component framework—problem definition, verification, validation, and continuous credibility building—with 16 concrete recommendations to guide rigorous reporting, data provenance, and reproducibility. Through qualitative discussion and real-world examples, the work highlights challenges such as data dependence, nonconvex optimization, and uncertain theory, while offering practical strategies for robust calibration, uncertainty quantification, and cross-domain evaluation. The contributions aim to enable trustworthy SciML across high-consequence domains by fostering transparency, standardization, and community-wide adoption of V&V practices that bridge CSE rigor with SciML innovation.

Abstract

Scientific machine learning (SciML) models are transforming many scientific disciplines. However, the development of good modeling practices to increase the trustworthiness of SciML has lagged behind its application, limiting its potential impact. The goal of this paper is to start a discussion on establishing consensus-based good practices for predictive SciML. We identify key challenges in applying existing computational science and engineering guidelines, such as verification and validation protocols, and provide recommendations to address these challenges. Our discussion focuses on predictive SciML, which uses machine learning models to learn, improve, and accelerate numerical simulations of physical systems. While centered on predictive applications, our 16 recommendations aim to help researchers conduct and document their modeling processes rigorously across all SciML domains.

Paper Structure

This paper contains 26 sections, 3 figures.

Table of Contents

  1. Introduction
  2. Background
  3. Scope
  4. Comparing CSE and SciML model development and deployment
  5. Leveraging an existing CSE framework for trustworthy SciML
  6. Problem definition
  7. Model purpose
  8. Verification, calibration, validation and application domain
  9. Quantities of interest
  10. Model conceptualization
  11. Verification
  12. Code verification
  13. Solution verification
  14. Validation
  15. Calibration
  16. ...and 11 more sections

Figures (3)

  • Figure 1: Fundamental steps for developing and deploying CSE and SciML models in outer-loop processes. Deployment phases are conceptually similar for both model types when making scientific claims. Simulation steps require multiple model runs based on outer-loop process demands.
  • Figure 2: Four components of trustworthy model development. Blue boxes indicate areas common to CSE and SciML models, while gray boxes show SciML-specific areas. Generative physics-informed SciML models may skip the gray data collection and processing boxes in code and solution verification.
  • Figure 3: Verification, validation, calibration and application domains.