Temporal Distribution Shift in Real-World Pharmaceutical Data: Implications for Uncertainty Quantification in QSAR Models

TL;DR

This study investigates uncertainty quantification for QSAR models under realistic distribution shifts by employing temporal splits on internal TB and ADME-T pharmaceutical assays. It compares train-time uncertainty methods (deep ensembles, MC dropout, Bayes-by-Backprop) with post hoc calibration (Platt scaling, Venn-ABERS) across 15 assays and three temporal settings. The results reveal pronounced shifts in label and descriptor spaces, especially for TB assays, which undermine calibration performance of several methods; deep ensembles and BNNs offer the best calibration for ADME-T datasets, while gains are limited for TB under large shifts. The work highlights the need for time-aware evaluation in real-world drug discovery and suggests practical guidance on method choice given shift magnitude and resource constraints.

Abstract

The estimation of uncertainties associated with predictions from quantitative structure-activity relationship (QSAR) models can accelerate the drug discovery process by identifying promising experiments and allowing an efficient allocation of resources. Several computational tools exist that estimate the predictive uncertainty in machine learning models. However, deviations from the i.i.d. setting have been shown to impair the performance of these uncertainty quantification methods. We use a real-world pharmaceutical dataset to address the pressing need for a comprehensive, large-scale evaluation of uncertainty estimation methods in the context of realistic distribution shifts over time. We investigate the performance of several uncertainty estimation methods, including ensemble-based and Bayesian approaches. Furthermore, we use this real-world setting to systematically assess the distribution shifts in label and descriptor space and their impact on the capability of the uncertainty estimation methods. Our study reveals significant shifts over time in both label and descriptor space and a clear connection between the magnitude of the shift and the nature of the assay. Moreover, we show that pronounced distribution shifts impair the performance of popular uncertainty estimation methods used in QSAR models. This work highlights the challenges of identifying uncertainty quantification methods that remain reliable under distribution shifts introduced by real-world data.

Figures (10)

• Figure 1: Overview of dataset sizes. The left panel plots the size of the individual assays ordered according to assay size. The striped areas in the bar indicate the amount of compounds belonging to the preferred class (PC) in each assay. The right panel shows the amount of training data in each temporal setting across all assays, with 1, 2, or 3 time spans used for training.
• Figure 2: Overview of the temporal split and model training. The data in each assay was assigned to 5 time spans to create three temporal settings, each with increasing amounts of training (Training) data. The subsequent two folds were used for validation (Valid.) and testing (Test). The validation data also served as a calibration set used in post hoc calibration approaches.
• Figure 3: Overview of the classification models. The architectures of the baseline models and train-time uncertainty quantification methods compared in this study are shown. All models were trained in a single-task manner. The hyperparameters of the baselines, RF and MLP, were tuned in an extensive grid search. The baseline MLP was used as the basis for the three uncertainty quantification methods, deep ensembles, MC dropout, and a Bayesian neural network.
• Figure 4: Quantification of the distribution shifts between the training and test datasets over time. The shift in label space and in the descriptor space is illustrated for each temporal setting, using the data of 1, 2, or 3 time spans for training. Results are shown for each assay. The left panel shows the shift in label space in terms of the difference in ratios of the preferred class between the training and test datasets. The right panel shows the MMD in the descriptor space between the training and test datasets for each temporal setting and assay, quantifying shifts in descriptor space.
• Figure 5: T-SNE plots of the ECFP space. T-SNE plots of the ECFP space are shown for one example of each assay category to illustrate how the explored chemical space changes over time. Compounds are colored according to the time span that they were assigned to. The t-SNE plot of the remaining TB and ADME-T assays are shown in Figures \ref{['fig_supp:tsne_TB']} and \ref{['fig_supp:tsne_ADMET']} in the appendix.