Can Large Language Models Infer Causal Relationships from Real-World Text?

TL;DR

ReCITE establishes a first real-world benchmark for inferring causal graphs from long-form academic texts, revealing that current LLMs struggle with causal reasoning beyond explicit statements. By assembling a diverse corpus, converting diagrams to text-based ground truth, and employing an LLM-as-a-Judge evaluation framework, the study provides targeted diagnostics on explicitness, graph size, and domain effects. The results show the best models achieving only 0.535 F1 on average, with substantial gaps between text-supported edges and ground-truth edges, indicating a fundamental limitation in causal inference from real-world narratives. The work also demonstrates a scalable, cost-efficient evaluation approach and confirms that human experts can perform the task, underscoring the potential for future progress and the need for improved modeling and evaluation methodologies.

Abstract

Understanding and inferring causal relationships from texts is a core aspect of human cognition and is essential for advancing large language models (LLMs) towards artificial general intelligence. Existing work evaluating LLM causal reasoning primarily relies on synthetic or simplified texts with explicitly stated causal relationships. These texts typically feature short passages and few causal relations, failing to reflect the complexities of real-world reasoning. In this paper, we investigate whether LLMs are capable of inferring causal relationships from real-world texts. We develop a benchmark drawn from real-world academic literature, which includes diverse texts with respect to length, complexity (different levels of explicitness, number of causal events and relationships), and domain. To the best of our knowledge, our benchmark is the first-ever real-world dataset for this task. Our experiments on this dataset show that LLMs face significant challenges in inferring causal relationships from real-world text, with the best-performing model achieving an average F$_1$ score of only 0.535. Through systematic analysis across aspects of real-world text (explicitness, number of causal events and relationships, length of text, domain), our benchmark offers targeted insights for further research into advancing LLM causal reasoning. Our code and dataset can be found at https://github.com/Ryan-Saklad/ReCITE .

Figures (17)

• Figure 1: An example causal graph illustrating the difference between explicit and implicit texts describing the same causal relationships. The explicit text directly states causal relationships using clear language ("directly improves," "leads to"). The implicit text describes the same relationships without stating causation ("spent more time training," "noticed improved attitudes"). This exemplifies a key challenge with real-world texts, where causal reasoning must be used to construct the graph.
• Figure 2: Heatmap of the average model scores across bins, where 100% is entirely explicit, and F$_1$ score from 0-1, showcasing how explicitness has a large impact on performance, and LLMs struggle to infer causality when explicit references are sparse.
• Figure 3: Radar chart of domain-specific accuracies for each model, depicting the accuracy of different models.
• Figure 4: Verbatim excerpts from R1's reasoning trace. Ellipses added for readability and bold for emphasis.
• Figure 5: Causal subgraph annotated with R1's performance. Green: concepts R1 correctly identified. Blue: ground-truth nodes not included. Dashed: ground-truth edges not included. Red: spurious nodes and edges.