CausalGraph2LLM: Evaluating LLMs for Causal Queries

TL;DR

This work introduces CausalGraph2LLM, a benchmark to evaluate how well LLMs encode and reason about causal DAGs across graph-level and node-level queries. It systematically compares seven encodings and multiple models, revealing substantial encoding-driven performance variation (up to about $60\%$) and biases arising from contextual knowledge. The study demonstrates that prompting strategy and graph representation critically shape LLM reasoning, with downstream intervention tasks also impacted by encoding and pretraining context. The benchmark provides a foundational resource for future work on robust, bias-aware causal reasoning with LLMs and informs practical encoding choices for causal tasks. Overall, it highlights the need for careful encoding selection, potential fine-tuning, and bias mitigation to responsibly deploy LLMs for causal inference tasks.

Abstract

Causality is essential in scientific research, enabling researchers to interpret true relationships between variables. These causal relationships are often represented by causal graphs, which are directed acyclic graphs. With the recent advancements in Large Language Models (LLMs), there is an increasing interest in exploring their capabilities in causal reasoning and their potential use to hypothesize causal graphs. These tasks necessitate the LLMs to encode the causal graph effectively for subsequent downstream tasks. In this paper, we introduce CausalGraph2LLM, a comprehensive benchmark comprising over 700k queries across diverse causal graph settings to evaluate the causal reasoning capabilities of LLMs. We categorize the causal queries into two types: graph-level and node-level queries. We benchmark both open-sourced and propriety models for our study. Our findings reveal that while LLMs show promise in this domain, they are highly sensitive to the encoding used. Even capable models like GPT-4 and Gemini-1.5 exhibit sensitivity to encoding, with deviations of about $60\%$. We further demonstrate this sensitivity for downstream causal intervention tasks. Moreover, we observe that LLMs can often display biases when presented with contextual information about a causal graph, potentially stemming from their parametric memory.

Figures (14)

• Figure 1: CausalGraph2LLM: Causal graphs are ingested into LLMs via prompt encoding strategies which are evaluated for causal queries.
• Figure 2: Different graph encoding functions for converting same causal graph to textual prompts, $p: G \rightarrow P$.
• Figure 3: An example prompt with single node encoding, with mediator graph-level query.
• Figure 4: Performance comparison across methods and encodings for graph-level queries.
• Figure 5: Performance of different models across (a) Insurance and (b) Alarm graphs. Bars represent performance without context, while dots indicate performance with context. The results are averaged over different encodings for each model.