Overconfident Oracles: Limitations of In Silico Sequence Design Benchmarking

TL;DR

This work questions the reliability of in silico biological sequence design benchmarks that rely on ML oracles by showing that method rankings vary with oracle seed and architecture, undermining cross-study comparisons. It demonstrates poor out-of-distribution generalization of common GFP/UTR oracles and proposes a biophysical validation suite to ground assessments in biological plausibility. The authors provide a concrete, data-driven path to more robust benchmarks by combining multiple oracles with task-relevant physical constraints, aiming to reduce wasted effort on implausible sequences and improve transfer to wet-lab success. Together, these contributions illuminate critical pitfalls in current evaluation pipelines and suggest practical steps to enhance the robustness and interpretability of in silico design methods.

Abstract

Machine learning methods can automate the in silico design of biological sequences, aiming to reduce costs and accelerate medical research. Given the limited access to wet labs, in silico design methods commonly use an oracle model to evaluate de novo generated sequences. However, the use of different oracle models across methods makes it challenging to compare them reliably, motivating the question: are in silico sequence design benchmarks reliable? In this work, we examine 12 sequence design methods that utilise ML oracles common in the literature and find that there are significant challenges with their cross-consistency and reproducibility. Indeed, oracles differing by architecture, or even just training seed, are shown to yield conflicting relative performance with our analysis suggesting poor out-of-distribution generalisation as a key issue. To address these challenges, we propose supplementing the evaluation with a suite of biophysical measures to assess the viability of generated sequences and limit out-of-distribution sequences the oracle is required to score, thereby improving the robustness of the design procedure. Our work aims to highlight potential pitfalls in the current evaluation process and contribute to the development of robust benchmarks, ultimately driving the improvement of in silico design methods.

Figures (7)

• Figure 1: Error between the oracle predictions and true dataset scores per sequence for the train (red) and held-out validation (blue) datasets for three GFP oracles: (left) Design-Bench transformer, (middle) TAPE, (right) ESM-1b fine-tuned on GFP dataset.
• Figure 2: TFBind-8: Absolute error between the Design Bench-inspired ML oracle predictions and ground truth dataset scores for train (red) and held-out validation (blue) datasplits.
• Figure 3: Generated sequences classified by DNA (left and centre) and protein (right) suite of biological measures, for 12 sequence design methods with respect to the task dataset (left and right) and the GENCODE database (centre).
• Figure 4: The distribution of scores for the UTR (left), GFP (centre), and TFBind-8 (right) datasets.
• Figure 5: Distribution of the maximum score in the batch of de novo sequences generated under 8 replications of the design methods and 3 different GFP oracles.