Beyond Perfect Scores: Proof-by-Contradiction for Trustworthy Machine Learning

TL;DR

This work analyzes trustworthiness in biomedical ML by addressing non-IID, hierarchical data. It introduces a Rubin causal-model-based framework that uses bucket-level label permutations and Fisher-style p-values to determine whether a model exploits genuine signal or dataset artefacts. The approach is validated on controlled benchmarks (Rotated MNIST, Colored FashionMNIST) and real biomedical data (QPM-WGS-AMR-21, Raman), showing that high accuracy alone can be misleading and that causal significance should accompany predictive performance. The framework is model- and dataset-agnostic, offers practical guidelines for efficient evaluation (pretraining and permutation counts), and aims to bridge ML with life sciences for more trustworthy clinical deployment.

Abstract

Machine learning (ML) models show strong promise for new biomedical prediction tasks, but concerns about trustworthiness have hindered their clinical adoption. In particular, it is often unclear whether a model relies on true clinical cues or on spurious hierarchical correlations in the data. This paper introduces a simple yet broadly applicable trustworthiness test grounded in stochastic proof-by-contradiction. Instead of just showing high test performance, our approach trains and tests on spurious labels carefully permuted based on a potential outcomes framework. A truly trustworthy model should fail under such label permutation; comparable accuracy across real and permuted labels indicates overfitting, shortcut learning, or data leakage. Our approach quantifies this behavior through interpretable Fisher-style p-values, which are well understood by domain experts across medical and life sciences. We evaluate our approach on multiple new bacterial diagnostics to separate tasks and models learning genuine causal relationships from those driven by dataset artifacts or statistical coincidences. Our work establishes a foundation to build rigor and trust between ML and life-science research communities, moving ML models one step closer to clinical adoption.

Figures (6)

• Figure 1: Representative papers with highly accurate deep learning models trained with small numbers of data buckets chung2023labelkim2022rapidlee2023machinesung2023threejo2017holographicahmad2023highlyhallstrom2025rapidli2021deepnissim2021realal2022highlyliu2023classificationliu2025deepzhou2022ramannet
• Figure 2: Overview of the proposed causal null-model framework for evaluating model trustworthiness through permutation-based stochastic proof-by-contradiction. Observed bucket-level labels $\mathbf{W}^{\text{obs}}$ serve as ground truth (GT). Random permutations $\{\mathbf{W}^{(m)}\}_{m=1}^M$ act as alternative ground truths to form the null distribution of test accuracies. The Fisher-style $p$-value quantifies whether the observed model performance $T_{\text{obs}}$ is significantly higher than those obtained under random label assignments.
• Figure 3: Simulated MNIST example with and without a causal signal on LightCNN. (a) Digits rotated by $45^\circ$ introduce a treatment cue, yielding significant separation between observed and null accuracies. (b) Without rotation, no true signal exists despite high apparent accuracy.
• Figure 4: Sample images Top: RGB versions of the images without any class-dependent color manipulation; Bottom: RGB images with class-dependent red scaling and shared green/blue baselines, altering global color without changing spatial structure.
• Figure 5: Representative Raman spectra for the MRSA vs. MSSA experiment. Each subclass (MRSA 1–2, MSSA 1–3) contains 100 spectra, with 1000 Raman shift points per spectrum. Spectral profiles between MRSA and MSSA appear visually similar, highlighting the challenge of discriminating antibiotic resistance from spectral data alone.