Efficient Biological Data Acquisition through Inference Set Design

TL;DR

Efficient Biological Data Acquisition through Inference Set Design tackles the cost of high-throughput screening by coordinating selective labeling and prediction on a fixed target set. The method, inference set design, uses a confidence-based least-confidence strategy to acquire labels for the hardest examples, leaving the easier cases for inference. A probabilistic stopping criterion based on a KL-divergence bound ensures the final system accuracy $μ_{sys}^t$ surpasses $γ$ with probability at least $1-δ$. Across MNIST variants, QM9, Molecules3D, and RxRx3, the approach achieves substantial experimental budget reductions while preserving or improving overall system performance, highlighting practical impact for drug discovery.

Abstract

In drug discovery, highly automated high-throughput laboratories are used to screen a large number of compounds in search of effective drugs. These experiments are expensive, so one might hope to reduce their cost by only experimenting on a subset of the compounds, and predicting the outcomes of the remaining experiments. In this work, we model this scenario as a sequential subset selection problem: we aim to select the smallest set of candidates in order to achieve some desired level of accuracy for the system as a whole. Our key observation is that, if there is heterogeneity in the difficulty of the prediction problem across the input space, selectively obtaining the labels for the hardest examples in the acquisition pool will leave only the relatively easy examples to remain in the inference set, leading to better overall system performance. We call this mechanism inference set design, and propose the use of a confidence-based active learning solution to prune out these challenging examples. Our algorithm includes an explicit stopping criterion that interrupts the acquisition loop when it is sufficiently confident that the system has reached the target performance. Our empirical studies on image and molecular datasets, as well as a real-world large-scale biological assay, show that active learning for inference set design leads to significant reduction in experimental cost while retaining high system performance.

Figures (13)

• Figure 1: Hybrid screen employing active learning as an inference set design strategy. The goal is to produce a "hybrid dataset", composed of the labels collected in the observation set and the predictions of the model made on the inference set. The system’s performance is reported as the "system accuracy", measured as a combination of observed labels on the observation set and of predictions made on the inference set. As samples are acquired and added to the observation set, the contribution of the accuracy of the prediction model gradually decreases. The aim of the active agent is to select informative examples while pruning out the most difficult ones in order to reach very high system accuracy without having to acquire all the samples.
• Figure 2: Performance of least-confidence (LC) and random acquisition functions on variations of MNIST (columns) across 3 seeds. a) Original: MNIST with its naturally occurring easy and hard examples. b) Partial Observability: MNIST where the bottom two thirds of the images have been cropped out. c) Noisy labeling function: MNIST with shuffled labels for images of 6's, 8's, and 9's. The system seeks a target accuracy $\gamma$ of $98\%$.
• Figure 3: Performance of active and random agents on MNIST subject to an increasing number of shuffled labels. As the task becomes harder, the accuracy gap increases, but the stopping time $\tau$ is triggered later.
• Figure 4: Percent of samples with shuffled labels acquired by each agent throughout the acquisition steps.
• Figure 5: Results on QM9. a) A sample of molecules from the QM9 dataset. b) Performance of active agents (LC and BALD), heuristic orderings, and random sampler on QM9 across 3 seeds.

Theorems & Definitions (2)

• Lemma 1
• proof