Clever Hans in Chemistry: Chemist Style Signals Confound Activity Prediction on Public Benchmarks

TL;DR

This work demonstrates a Clever Hans–style shortcut in public medicinal chemistry benchmarks: models can predict bioactivity largely by inferring chemist goals from structure, not by learning true structure–activity relationships. By linking ChEMBL assays to 1,815 authors, the authors show that a simple fingerprint-based classifier can identify the responsible chemist from structure with notable accuracy, and that an activity model using only author-probabilities and protein identity achieves performance close to structure-based baselines. The results reveal that provenance signals and lab-specific styles can confound benchmark interpretation, and they advocate author-aware splits and provenance-aware baselines to decouple chemist intent from biology. These findings underscore the need for better dataset practices and evaluation design to ensure that reported activity predictions reflect genuine chemistry rather than data provenance shortcuts.

Abstract

Can machine learning models identify which chemist made a molecule from structure alone? If so, models trained on literature data may exploit chemist intent rather than learning causal structure-activity relationships. We test this by linking CHEMBL assays to publication authors and training a 1,815-class classifier to predict authors from molecular fingerprints, achieving 60% top-5 accuracy under scaffold-based splitting. We then train an activity model that receives only a protein identifier and an author-probability vector derived from structure, with no direct access to molecular descriptors. This author-only model achieves predictive power comparable to a simple baseline that has access to structure. This reveals a "Clever Hans" failure mode: models can predict bioactivity largely by inferring chemist goals and favorite targets without requiring a lab-independent understanding of chemistry. We analyze the sources of this leakage, propose author-disjoint splits, and recommend dataset practices to decouple chemist intent from biological outcomes.

Figures (8)

• Figure 1: Overview of the two-stage chemist style leakage test. Stage 1: We train a model to predict which of 1,815 authors synthesized a molecule from its structure alone, achieving 60% top-5 accuracy. Stage 2: We use these author probabilities (plus protein ID) to predict activity, without providing molecular features. The strong performance (validation AUROC around 0.65) reveals a shortcut: models can predict activity by inferring chemist intent without explicitly modeling structure--activity relationships.
• Figure 2: Classic example of a Clever Hans predictor, reproduced from Lapuschkin et al. lapuschkin2019cleverhans. The left panel shows an image correctly classified as containing a horse; the right panel shows a relevance heatmap revealing that the classifier bases its decision primarily on the photographer’s corner watermark rather than on the horse itself. By analogy, in this work we argue that molecules in ChEMBL carry chemist-specific “watermarks” in their structure, which even simple models can exploit, confounding attempts to learn genuine causes of binding
• Figure 3: Histogram of ROC AUC scores of each author in one-vs-rest models.
• Figure 4: Validation performance of different feature sets under an identical GBDT classifier. Bars show mean performance over five random scaffold splits; error bars denote one standard deviation. The author + protein model tracks the ECFP + protein model close in AUROC, modest gap in AP. Combining author probabilities with ECFPs yields only modest additional gains, indicating that a large fraction of the predictive signal is already present in chemist style and target identity.
• Figure 5: Five exemplar molecules from Carrie Haskell-Luevano and the corresponding author-confusion profile. The top panel shows macrocyclic melanocortin ligands built from densely functionalized, noncanonical amino acids and conformational constraints that are rare elsewhere in ChEMBL. These macrocycle-heavy series make Haskell-Luevano the most stylistically identifiable chemist in our prolific-author cohort: the author classifier assigns her as the top prediction for 75.5% of held-out molecules under scaffold splitting. The bottom panel shows the confusion-matrix row for this author, with a sharp diagonal peak and only limited spillover onto a small number of chemically similar peptide labs.