Document-level Causal Relation Extraction with Knowledge-guided Binary Question Answering

TL;DR

The paper tackles document-level ECRE by addressing document-level semantic gaps and causal hallucinations in LLMs through KnowQA, a two-stage method that builds rich, document-wide event structures and then solves ECRE as binary QA with single-turn and multi-turn prompts. It shows that incorporating complete event structures significantly improves document-level ECRE, achieving state-of-the-art results on MECI and strong generalization on MAVEN-ERE, especially after model fine-tuning. The approach demonstrates high robustness to cross-task knowledge and variations in causal phrasing, while also offering an inconsistency metric that drops substantially with complete structures. Overall, KnowQA advances reliable document-level causal reasoning and provides a framework for extending to additional relations and multilingual settings with strong potential for downstream IE tasks.

Abstract

As an essential task in information extraction (IE), Event-Event Causal Relation Extraction (ECRE) aims to identify and classify the causal relationships between event mentions in natural language texts. However, existing research on ECRE has highlighted two critical challenges, including the lack of document-level modeling and causal hallucinations. In this paper, we propose a Knowledge-guided binary Question Answering (KnowQA) method with event structures for ECRE, consisting of two stages: Event Structure Construction and Binary Question Answering. We conduct extensive experiments under both zero-shot and fine-tuning settings with large language models (LLMs) on the MECI and MAVEN-ERE datasets. Experimental results demonstrate the usefulness of event structures on document-level ECRE and the effectiveness of KnowQA by achieving state-of-the-art on the MECI dataset. We observe not only the effectiveness but also the high generalizability and low inconsistency of our method, particularly when with complete event structures after fine-tuning the models.

Figures (5)

• Figure 1: Overview of the ECRE process. The dashed lines indicate there are no causal relationships between the event mentions.
• Figure 2: Overall framework of the proposed KnowQA method for ECRE, consisting of two stages: Event Structure Construction and Binary Question Answering. In the first stage, we utilize IE models to form event structures at the document level. In the second stage, we formulate ECRE as a binary QA task with single-turn and multi-turn strategies and fully leverage the event structures for ECRE predictions.
• Figure 3: Event structures for the example shown in Figure \ref{['fig:framework']}, consisting of event mentions, event arguments, and the single-hop relationships of the arguments.
• Figure 4: Performance of $\text{Flan-T5}_\textit{Large}$ with complete event structures using different causal expressions on the MECI dataset.
• Figure 5: Inconsistency evaluation results of the models on the MECI dataset.