Causal Question Answering with Reinforcement Learning

TL;DR

This paper introduces an Actor-Critic-based agent which learns to search through the graph to answer causal questions with a causality graph, a large-scale dataset of causal relations between noun phrases along with the relations' provenance data.

Abstract

Causal questions inquire about causal relationships between different events or phenomena. They are important for a variety of use cases, including virtual assistants and search engines. However, many current approaches to causal question answering cannot provide explanations or evidence for their answers. Hence, in this paper, we aim to answer causal questions with a causality graph, a large-scale dataset of causal relations between noun phrases along with the relations' provenance data. Inspired by recent, successful applications of reinforcement learning to knowledge graph tasks, such as link prediction and fact-checking, we explore the application of reinforcement learning on a causality graph for causal question answering. We introduce an Actor-Critic-based agent which learns to search through the graph to answer causal questions. We bootstrap the agent with a supervised learning procedure to deal with large action spaces and sparse rewards. Our evaluation shows that the agent successfully prunes the search space to answer binary causal questions by visiting less than 30 nodes per question compared to over 3,000 nodes by a naive breadth-first search. Our ablation study indicates that our supervised learning strategy provides a strong foundation upon which our reinforcement learning agent improves. The paths returned by our agent explain the mechanisms by which a cause produces an effect. Moreover, for each edge on a path, our causality graph provides its original source allowing for easy verification of paths.

Figures (4)

• Figure 1: An excerpt from CauseNet Heindorf2020Causenet showing the entity pneumonia together with its neighborhood containing causes and effects, where each edge depicts a cause relation. The numbers on the edges show the probability of taking this edge under the current policy $\mathbf{\pi_{\theta}}(a_t | s_t)$. For brevity, we only show the relevant probabilities for the given paths. The lower part of the figure shows the possibility to combine our agent with a language model. In that setup, we provide the paths the agent learned as additional context to the language model.
• Figure 2: Accuracy of the agent on the SemEval test set depending on the number of reinforcement learning training steps. Each run was bootstrapped with a different number of supervised training steps. Step 0 shows the performance directly after supervised learning.
• Figure 3: Number of unique paths explored during reinforcement learning training on the left and the mean entropy of the action distribution of the policy network on the right. Each run was bootstrapped with a different number of supervised training steps.
• Figure 4: Accuracy on the MS MARCO test set depending on the number of RL training steps and beam width.