Beyond validation loss: Clinically-tailored optimization metrics improve a model's clinical performance

TL;DR

This work argues that clinically-tailored optimization metrics, rather than traditional validation loss, better align ML model development with real-world medical needs. It demonstrates this through two real-world experiments: (1) hyperparameter optimization driven by patient-level versus object-level FoMs in a Loa loa detection task, showing superior patient-level performance when optimizing clinically-relevant metrics; and (2) stopping-point selection for a DNN diagnosing twins vs. singletons using multiple FoMs such as $90\%$ sliver AUC and sensitivity at $90\%$ specificity, revealing later, more clinically robust stopping points than those suggested by validation loss. The paper emphasizes the bespoke nature of FoM design, the need for collaboration with clinicians, and practical methods (e.g., z-scale alignment) to integrate these metrics into optimization workflows. Collectively, the findings advocate for embedding clinically-relevant FoMs early in model development to maximize in-clinic utility and robustness.

Abstract

A key task in ML is to optimize models at various stages, e.g. by choosing hyperparameters or picking a stopping point. A traditional ML approach is to use validation loss, i.e. to apply the training loss function on a validation set to guide these optimizations. However, ML for healthcare has a distinct goal from traditional ML: Models must perform well relative to specific clinical requirements, vs. relative to the loss function used for training. These clinical requirements can be captured more precisely by tailored metrics. Since many optimization tasks do not require the driving metric to be differentiable, they allow a wider range of options, including the use of metrics tailored to be clinically-relevant. In this paper we describe two controlled experiments which show how the use of clinically-tailored metrics provide superior model optimization compared to validation loss, in the sense of better performance on the clinical task. The use of clinically-relevant metrics for optimization entails some extra effort, to define the metrics and to code them into the pipeline. But it can yield models that better meet the central goal of ML for healthcare: strong performance in the clinic.

Figures (9)

• Figure 1: Frames of blood capillary videos, normal (left) and very coagulated (right). The dataset contains a range of coagulations.
• Figure 2: Best ROC curves for patient-level (red) and video-level (black). Per subplot, the two curves are from the best hyperopt iteration, as driven by: (left) object-level AUC, (right) patient-level AUC. Patient-level optimization gives much better patient-level performance.
• Figure 3: Patient-level and video-level AUC values for different iterations of hyperopt. x-axis = sorted iteration index, y-axis = AUC value. Patient-level AUCs in green, video-level AUCs in blue. A: Optimizer driven by patient-level AUC, iterations sorted by patient-level AUC values. B: optimizer driven by video-level AUC, iterations sorted by video-level AUC values. C: Scatterplot of patient-level vs video-level AUCs for data in subplot B. Note the lack of correlation between the two performance metrics.
• Figure 4: Sliver AUC example: The blue ROC has lower overall AUC than the red (0.9 vs 0.96), but it is better at high specificities and thus has a higher 90% sliver AUC (in the grey column, 0.82 vs 0.77).
• Figure 5: Figure of Merit time-series over training epochs. x-axis: epoch indices, y-axis: FoM value. L - R: (1) -1 * Standard validation loss; (2) AUC; (3) 90% Sliver AUC; (4) Sensitivity at 90% specificity; (5) Fisher distance. x-axis: epoch. y-axis: value (higher is better). Highest scores marked in red. Clinically-relevant FoMs suggest much later stopping points than that of validation loss.