Case-Guided Sequential Assay Planning in Drug Discovery

TL;DR

This work tackles sequential assay planning in drug discovery under simulator-free uncertainty and budget constraints. It introduces IBMDP, a model-based RL framework that builds an implicit, case-based transition model from historical outcomes via similarity-weighted sampling and performs ensemble MCTS planning to optimize multi-step experimental plans. Key contributions include a similarity-weighted Bayesian belief mechanism, nonparametric implicit dynamics, and robust ensemble planning that yields the MLASP policy, demonstrated to dramatically reduce resource use (up to 92%) while preserving decision confidence on a real CNS brain-penetration task and showing favorable alignment with a computable optimal policy in synthetic benchmarks. The approach enables principled, data-efficient experimentation in data-rich but simulator-poor domains, with potential impact across discovery sciences where explicit mechanistic models are unavailable.

Abstract

Optimally sequencing experimental assays in drug discovery is a high-stakes planning problem under severe uncertainty and resource constraints. A primary obstacle for standard reinforcement learning (RL) is the absence of an explicit environment simulator or transition data $(s, a, s')$; planning must rely solely on a static database of historical outcomes. We introduce the Implicit Bayesian Markov Decision Process (IBMDP), a model-based RL framework designed for such simulator-free settings. IBMDP constructs a case-guided implicit model of transition dynamics by forming a nonparametric belief distribution using similar historical outcomes. This mechanism enables Bayesian belief updating as evidence accumulates and employs ensemble MCTS planning to generate stable policies that balance information gain toward desired outcomes with resource efficiency. We validate IBMDP through comprehensive experiments. On a real-world central nervous system (CNS) drug discovery task, IBMDP reduced resource consumption by up to 92\% compared to established heuristics while maintaining decision confidence. To rigorously assess decision quality, we also benchmarked IBMDP in a synthetic environment with a computable optimal policy. Our framework achieves significantly higher alignment with this optimal policy than a deterministic value iteration alternative that uses the same similarity-based model, demonstrating the superiority of our ensemble planner. IBMDP offers a practical solution for sequential experimental design in data-rich but simulator-poor domains.

Figures (5)

• Figure 1: Sequential decisions in drug discovery through a data-driven, analog-guided simulator for planning, which maintains a Bayesian belief over the most relevant historical compound analogs.
• Figure 2: Monetary-prioritized results from IBMDP for four representative compounds. Each plot shows the Pareto front of achievable resource consumption versus terminal state uncertainty, with the Maximum-Likelihood Action-Sets Path (MLASP) highlighted. This illustrates how IBMDP provides a trade-off curve, allowing decision-makers to select a plan based on their risk and budget tolerance.
• Figure 3: Example histogram of actions proposed across an ensemble of $N_e=50$ runs. For a given state with uncertainty $\mathcal{H}(s) = 0.2$ and a likelihood constraint of $\tau = 0.9$, the action with the highest frequency is selected for the MLASP. This demonstrates how the ensemble method produces robust and stable recommendations via majority voting.
• Figure 4: Comparison of MLASP paths for the same compound under two different goal-likelihood thresholds: $\tau=0.6$ (blue) and $\tau=0.9$ (red). The stricter constraint ($\tau=0.9$) forces the planner to recommend a more expensive sequence of assays to achieve higher confidence, illustrating the direct trade-off between cost and decision confidence controlled by this parameter.
• Figure 5: ADME clearance optimization results comparing IBMDP performance under two belief thresholds ($\tau = 0.6$ and $\tau = 0.9$) for a representative compound from the public CNS clearance benchmark. The plot demonstrates the Pareto-optimal trade-offs between total assay spend (horizontal axis) and terminal state uncertainty $H(s_T)$ (vertical axis) achieved by the IBMDP ensemble across 30 runs. The two distinct curves for $\tau = 0.6$ (more lenient) and $\tau = 0.9$ (more stringent) illustrate how tighter belief thresholds drive higher assay expenditure to achieve lower uncertainty. Notably, the two tau configurations exhibit strong alignment in their Pareto frontiers, confirming that IBMDP produces consistent and robust planning strategies across different confidence requirements. The Maximum-Likelihood Action-Set Paths (MLASPs) for each threshold are marked, showing how the ensemble consensus adapts to balance the high cost of human clearance assays ($4,000) against the need to reduce decision uncertainty below the specified threshold.