How Well Do LLMs Understand Drug Mechanisms? A Knowledge + Reasoning Evaluation Dataset
Sunil Mohan, Theofanis Karaletsos
TL;DR
The paper introduces the Drug Mechanisms Counterfactuals Dataset to evaluate whether large language models can recall factual drug-mechanism knowledge and perform reasoning about MoA graphs in novel scenarios. It formalizes MoA graphs, defines directional consistency for graph comparison, and provides two datasets: one for factual MoA knowledge and a counterfactual dataset with Add-Link, Delete-Link, and Invert-Link perturbations, including deep (internal) and surface (drug-adjacent) variants. Across five OpenAI models and Qwen3-4B, the authors show that open-world reasoning is more demanding than closed-world prompting, with deep counterfactuals posing the greatest challenge; smaller, reasoning-focused models like o4-mini and Qwen3-4B can match or exceed larger models in many settings. The work highlights the need for updated knowledge bases and careful combination of LLMs with structured biomedical knowledge to reliably reason about drug mechanisms in real-world research contexts.
Abstract
Two scientific fields showing increasing interest in pre-trained large language models (LLMs) are drug development / repurposing, and personalized medicine. For both, LLMs have to demonstrate factual knowledge as well as a deep understanding of drug mechanisms, so they can recall and reason about relevant knowledge in novel situations. Drug mechanisms of action are described as a series of interactions between biomedical entities, which interlink into one or more chains directed from the drug to the targeted disease. Composing the effects of the interactions in a candidate chain leads to an inference about whether the drug might be useful or not for that disease. We introduce a dataset that evaluates LLMs on both factual knowledge of known mechanisms, and their ability to reason about them under novel situations, presented as counterfactuals that the models are unlikely to have seen during training. Using this dataset, we show that o4-mini outperforms the 4o, o3, and o3-mini models from OpenAI, and the recent small Qwen3-4B-thinking model closely matches o4-mini's performance, even outperforming it in some cases. We demonstrate that the open world setting for reasoning tasks, which requires the model to recall relevant knowledge, is more challenging than the closed world setting where the needed factual knowledge is provided. We also show that counterfactuals affecting internal links in the reasoning chain present a much harder task than those affecting a link from the drug mentioned in the prompt.