Benchmarking LLMs for Pairwise Causal Discovery in Biomedical and Multi-Domain Contexts
Sydney Anuyah, Sneha Shajee-Mohan, Ankit-Singh Chauhan, Sunandan Chakraborty
TL;DR
The paper presents a unified, large-scale benchmark to assess open-source LLMs on pairwise causal discovery from text, focusing separately on causal detection and causal extraction across 12 diverse datasets. By systematically varying prompts (zero-shot, in-context learning, chain-of-thought, and hybrids) and evaluating models from 3B to 70B, the authors reveal persistent gaps in current LLMs, especially for implicit and inter-sentential causality in biomedical and multi-domain texts. The study establishes a rigorous annotation protocol with high inter-annotator agreement and provides all data, prompts, and code to spur future work. Key findings show that instruction-tuned models generally perform better, but no model excels across all tasks, underscoring the need for domain-specific fine-tuning and integration with external knowledge graphs for safe, reliable causal reasoning in healthcare. The work furnishes a reproducible evaluation framework and quantitative insights to guide the development of more robust causal reasoning capabilities in LLMs for high-stakes domains.
Abstract
The safe deployment of large language models (LLMs) in high-stakes fields like biomedicine, requires them to be able to reason about cause and effect. We investigate this ability by testing 13 open-source LLMs on a fundamental task: pairwise causal discovery (PCD) from text. Our benchmark, using 12 diverse datasets, evaluates two core skills: 1) \textbf{Causal Detection} (identifying if a text contains a causal link) and 2) \textbf{Causal Extraction} (pulling out the exact cause and effect phrases). We tested various prompting methods, from simple instructions (zero-shot) to more complex strategies like Chain-of-Thought (CoT) and Few-shot In-Context Learning (FICL). The results show major deficiencies in current models. The best model for detection, DeepSeek-R1-Distill-Llama-70B, only achieved a mean score of 49.57\% ($C_{detect}$), while the best for extraction, Qwen2.5-Coder-32B-Instruct, reached just 47.12\% ($C_{extract}$). Models performed best on simple, explicit, single-sentence relations. However, performance plummeted for more difficult (and realistic) cases, such as implicit relationships, links spanning multiple sentences, and texts containing multiple causal pairs. We provide a unified evaluation framework, built on a dataset validated with high inter-annotator agreement ($κ\ge 0.758$), and make all our data, code, and prompts publicly available to spur further research. \href{https://github.com/sydneyanuyah/CausalDiscovery}{Code available here: https://github.com/sydneyanuyah/CausalDiscovery}