$\text{C}^2\text{P}$: Featuring Large Language Models with Causal Reasoning

TL;DR

The paper tackles the challenge that LLMs struggle with true causal reasoning by introducing $ ext{C}^2 ext{P}$, an autonomous chain-of-prompting framework that builds a causal adjacency structure (PDAG-like) from premises without external tools. It formalizes a five-subtask pipeline to extract variables and dependencies, refine a dense adjacency into causal structure, and pose hypotheses, enabling LLMs to perform reasoning aligned with Pearl's causal framework. Across synthetic and real-world benchmarks, including CORR2CAUSE, Natural Stories, and coevolution scenarios for supermassive black holes, $ ext{C}^2 ext{P}$ yields substantial improvements over baseline prompting methods and demonstrates strong few-shot generalization on GPT-4 Turbo and LLaMA 3.1. The work highlights the potential of autonomous causal prompting to elevate reasoning capabilities in LLMs, while noting identifiability and prompt-design challenges that guide future improvements and potential fine-tuning to scale the approach.

Abstract

Causal reasoning is one of the primary bottlenecks that Large Language Models (LLMs) must overcome to attain human-level intelligence. Recent studies indicate that LLMs display near-random performance on reasoning tasks. To address this, we introduce the Causal Chain of Prompting ($\text{C}^2\text{P}$), a reasoning framework that aims to equip current LLMs with causal reasoning capabilities as the first framework of its kind operating autonomously without relying on external tools or modules during both the causal learning and reasoning phases. To evaluate the performance of $\text{C}^2\text{P}$, we first demonstrate that reasoning accuracy improved by over $30.7\%$ and $25.9\%$ for GPT-4 Turbo and LLaMA 3.1, respectively, when using our framework, compared to the same models without $\text{C}^2\text{P}$ on a synthetic benchmark dataset. Then, using few-shot learning of the same LLMs with $\text{C}^2\text{P}$, the reasoning accuracy increased by more than $20.05\%$ and $20.89\%$, respectively, with as few as ten examples, compared to the corresponding LLMs without $\text{C}^2\text{P}$ on the same dataset. To evaluate $\text{C}^2\text{P}$ in realistic scenarios, we utilized another benchmark dataset containing natural stories across various fields, including healthcare, medicine, economics, education, social sciences, environmental science, and marketing. The results show improved reasoning when $\text{C}^2\text{P}$ is applied, compared to cases where our framework is not used, which often leads to random and hallucinated responses. By showing the improved performance of few-shot learned GPT-4 Turbo and LLaMA 3.1 with $\text{C}^2\text{P}$, we demonstrate the generalizability of our framework.

Figures (4)

• Figure 1: The 5 successive subtasks of $\text{C}^2\text{P}$ applied to Black Holes and their Host Galexis example presented in pasquato2023causa. Central Density(CD), Black Hole Mass(BHM), Velocity Density(VD), Bulge Stellar Mass(BSM), Effective Radius(ER)
• Figure 2: All possible in cause and effect relations. i. $X_1$ directly causes $X_2$ ($X_2$ is directly effect of $X_1$). ii. $X_1$ indirectly causes $X_3$ ($X_3$ is indirectly effect of $X_1$). iii. $X_1$ and $X_3$ are causes of $X_2$ ($X_2$ is common effect of $X_1$ and $X_3$)
• Figure 3: The accuracy metrics for different numbers of samples with 3 different methods.
• Figure 4: Prompts (Q) and results (A) of subtasks application of the $\text{C}^2\text{P}$ framework to real-world complex scenarios and steps of subtask 3 for the given premise.