Exploring Defeasibility in Causal Reasoning

TL;DR

This work presents $\delta$-CAUSAL, the first benchmark dataset for studying defeasibility in causal reasoning, and proposes CESAR (Causal Embedding aSsociation with Attention Rating), a metric that measures causal strength based on token-level causal relationships.

Abstract

Defeasibility in causal reasoning implies that the causal relationship between cause and effect can be strengthened or weakened. Namely, the causal strength between cause and effect should increase or decrease with the incorporation of strengthening arguments (supporters) or weakening arguments (defeaters), respectively. However, existing works ignore defeasibility in causal reasoning and fail to evaluate existing causal strength metrics in defeasible settings. In this work, we present $δ$-CAUSAL, the first benchmark dataset for studying defeasibility in causal reasoning. $δ$-CAUSAL includes around 11K events spanning ten domains, featuring defeasible causality pairs, i.e., cause-effect pairs accompanied by supporters and defeaters. We further show current causal strength metrics fail to reflect the change of causal strength with the incorporation of supporters or defeaters in $δ$-CAUSAL. To this end, we propose CESAR (Causal Embedding aSsociation with Attention Rating), a metric that measures causal strength based on token-level causal relationships. CESAR achieves a significant 69.7% relative improvement over existing metrics, increasing from 47.2% to 80.1% in capturing the causal strength change brought by supporters and defeaters. We further demonstrate even Large Language Models (LLMs) like GPT-3.5 still lag 4.5 and 10.7 points behind humans in generating supporters and defeaters, emphasizing the challenge posed by $δ$-CAUSAL.

Figures (8)

• Figure 1: A motivational example of defeasibility in causal reasoning. It consists of a cause-effect pair, a supporting argument (supporter), and an opposing argument (defeater) for the causal relationship.
• Figure 2: The overview of the causal relationships in delta-Causal. The symbol $\textbf{+}$ indicates that the supporter $A$ strengthens the causal relationship between the cause $C$ and the effect $E$, while -- signifies that the defeater $D$ weakens this relationship. The variable $\Delta_t$ denotes the time interval between the cause $C$ and the effect $E$. To ensure the practicality of identifying defeaters, this time interval is set to long time durations.
• Figure 3: Pipeline of the annotation and refinement procedures of delta-Causal.
• Figure 4: The distributions of time intervals and domains in delta-Causal. Different colors represent different time intervals (inner circle) and domains (outer circle). Detailed values and proportions are provided in Appendix \ref{['appendix:statistics:time_interval']}.
• Figure 5: The shifts in causal strength distributions facilitated by CEQ (left), ROCK (middle), and CESAR (right) with the incorporation of supporters and defeaters are illustrated in delta-Causal. These curves utilize kernel density estimation parzen1962estimation to depict the data distribution as a continuous probability density curve. Notably, only CESAR effectively captures the variations in causal strength triggered by the inclusion of supporters and defeaters; specifically, the causal strength distribution shifts to the right with supporters and to the left with defeaters.