Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Assessing LLM Reasoning Through Implicit Causal Chain Discovery in Climate Discourse

Liesbeth Allein, Nataly Pineda-Castañeda, Andrea Rocci, Marie-Francine Moens

TL;DR

The paper tackles the problem of evaluating mechanistic causal reasoning in LLMs by asking models to generate implicit causal chains linking cause-effect pairs in polarized climate discourse. It introduces a zero-shot causal chain discovery approach, evaluates nine LLMs on data from PolarIs3CAUS and PolarIs4CAUS, and validates results with automatic metrics and human expert judgments. Key findings show that while models are self-consistent and confident about intermediate links, their judgments largely rely on associative patterns rather than true causal inference, and directionality framing significantly affects conclusions. The work provides a baseline methodology, a benchmark dataset of chains, and insights for building more robust, mechanistic reasoning in argumentation contexts.

Abstract

How does a cause lead to an effect, and which intermediate causal steps explain their connection? This work scrutinizes the mechanistic causal reasoning capabilities of large language models (LLMs) to answer these questions through the task of implicit causal chain discovery. In a diagnostic evaluation framework, we instruct nine LLMs to generate all possible intermediate causal steps linking given cause-effect pairs in causal chain structures. These pairs are drawn from recent resources in argumentation studies featuring polarized discussion on climate change. Our analysis reveals that LLMs vary in the number and granularity of causal steps they produce. Although they are generally self-consistent and confident about the intermediate causal connections in the generated chains, their judgments are mainly driven by associative pattern matching rather than genuine causal reasoning. Nonetheless, human evaluations confirmed the logical coherence and integrity of the generated chains. Our baseline causal chain discovery approach, insights from our diagnostic evaluation, and benchmark dataset with causal chains lay a solid foundation for advancing future work in implicit, mechanistic causal reasoning in argumentation settings.

Assessing LLM Reasoning Through Implicit Causal Chain Discovery in Climate Discourse

TL;DR

The paper tackles the problem of evaluating mechanistic causal reasoning in LLMs by asking models to generate implicit causal chains linking cause-effect pairs in polarized climate discourse. It introduces a zero-shot causal chain discovery approach, evaluates nine LLMs on data from PolarIs3CAUS and PolarIs4CAUS, and validates results with automatic metrics and human expert judgments. Key findings show that while models are self-consistent and confident about intermediate links, their judgments largely rely on associative patterns rather than true causal inference, and directionality framing significantly affects conclusions. The work provides a baseline methodology, a benchmark dataset of chains, and insights for building more robust, mechanistic reasoning in argumentation contexts.

Abstract

How does a cause lead to an effect, and which intermediate causal steps explain their connection? This work scrutinizes the mechanistic causal reasoning capabilities of large language models (LLMs) to answer these questions through the task of implicit causal chain discovery. In a diagnostic evaluation framework, we instruct nine LLMs to generate all possible intermediate causal steps linking given cause-effect pairs in causal chain structures. These pairs are drawn from recent resources in argumentation studies featuring polarized discussion on climate change. Our analysis reveals that LLMs vary in the number and granularity of causal steps they produce. Although they are generally self-consistent and confident about the intermediate causal connections in the generated chains, their judgments are mainly driven by associative pattern matching rather than genuine causal reasoning. Nonetheless, human evaluations confirmed the logical coherence and integrity of the generated chains. Our baseline causal chain discovery approach, insights from our diagnostic evaluation, and benchmark dataset with causal chains lay a solid foundation for advancing future work in implicit, mechanistic causal reasoning in argumentation settings.
Paper Structure (24 sections, 2 equations, 3 figures, 4 tables)

This paper contains 24 sections, 2 equations, 3 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Contributions
  3. Causal Chain Discovery
  4. Task Description
  5. Data
  6. Language Models
  7. Methodology
  8. General Results
  9. Diagnostic Evaluation
  10. Inside Causal Chains: Intermediate CE pairs
  11. Self-Consistency and Confidence in Causality (A1)
  12. -- Results
  13. Directionality of Causality (A2)
  14. -- Results
  15. Fallacy in Causal Reasoning: Position Heuristics (A3)
  16. ...and 9 more sections

Figures (3)

  • Figure 1: This paper proposes the implicit causal chain discovery task for revealing multiple intermediate causal mechanisms (b) underlying a cause-effect relation that is put forward in an argumentative setting (a). It analyzes the chains and their intermediate causal relations state-of-the-art language models reconstruct and that explain the different latent causal pathways existing between an initial cause and final effect.
  • Figure 2: Cross-model evaluation of the integrity of the chains generated for the CE pairs from PolarIs4CAUS. The results for the A1 setup are shown on top, those for the A2 setup below. In both figures, the LLMs that generated the chains are on the x-axis, the LLMs that evaluate those chains are on the y-axis, and the values report the proportion of chains that are considered valid by the LLM.
  • Figure 3: Pearson correlation between chain length and the proportion (%) of causal intermediate CE pairs (a), and between the number of chains per CE pair and the proportion (%) of causal intermediate CE pairs (b). The y-axis includes models with significant correlation in their generations ($p < .01$). The x-axis shows the range of $r$ values. Causality of pairs is evaluated using $\lambda_{A1}$ and $\lambda_{A2}$.