CHECKWHY: Causal Fact Verification via Argument Structure

TL;DR

CheckWhy is introduced, a challenging dataset tailored to a novel causal fact verification task: checking the truthfulness of the causal relation within claims through rigorous reasoning steps through extensive experiments on state-of-the-art models.

Abstract

With the growing complexity of fact verification tasks, the concern with "thoughtful" reasoning capabilities is increasing. However, recent fact verification benchmarks mainly focus on checking a narrow scope of semantic factoids within claims and lack an explicit logical reasoning process. In this paper, we introduce CheckWhy, a challenging dataset tailored to a novel causal fact verification task: checking the truthfulness of the causal relation within claims through rigorous reasoning steps. CheckWhy consists of over 19K "why" claim-evidence-argument structure triplets with supports, refutes, and not enough info labels. Each argument structure is composed of connected evidence, representing the reasoning process that begins with foundational evidence and progresses toward claim establishment. Through extensive experiments on state-of-the-art models, we validate the importance of incorporating the argument structure for causal fact verification. Moreover, the automated and human evaluation of argument structure generation reveals the difficulty in producing satisfying argument structure by fine-tuned models or Chain-of-Thought prompted LLMs, leaving considerable room for future improvements.

Figures (13)

• Figure 1: An entry from CheckWhy: a "why" claim with its corresponding cause and effect, and an argument structure representing the reasoning process from cause to effect. Notably, the cause-effect pair is used solely during the annotation process and not included in the argument structure, implying that it is implicitly inferred from the claim, rather than being provided explicitly.
• Figure 2: The human-model collaboration annotation process of CheckWhy, which contains three steps: (I) data preparation; (II) generation with LLMs (including the generation of claim, evidence, and argument structure); (III) manual validation.
• Figure 3: Histogram of useful evidence number in the CheckWhy
• Figure 4: Histogram of argument steps in the CheckWhy
• Figure 5: Types of Argument Structure