Plausible Extractive Rationalization through Semi-Supervised Entailment Signal

TL;DR

This paper takes a semi-supervised approach to optimize for the plausibility of extracted rationales, adopting a pre-trained natural language inference model and further fine-tune it on a small set of supervised rationales.

Abstract

The increasing use of complex and opaque black box models requires the adoption of interpretable measures, one such option is extractive rationalizing models, which serve as a more interpretable alternative. These models, also known as Explain-Then-Predict models, employ an explainer model to extract rationales and subsequently condition the predictor with the extracted information. Their primary objective is to provide precise and faithful explanations, represented by the extracted rationales. In this paper, we take a semi-supervised approach to optimize for the plausibility of extracted rationales. We adopt a pre-trained natural language inference (NLI) model and further fine-tune it on a small set of supervised rationales ($10\%$). The NLI predictor is leveraged as a source of supervisory signals to the explainer via entailment alignment. We show that, by enforcing the alignment agreement between the explanation and answer in a question-answering task, the performance can be improved without access to ground truth labels. We evaluate our approach on the ERASER dataset and show that our approach achieves comparable results with supervised extractive models and outperforms unsupervised approaches by $> 100\%$.

Figures (4)

• Figure 1: An example from the FEVER dataset, where the bold statement is the annotated rationale. Given the document and claim, the label denotes that the document contains evidence supporting the claim. The NLI predictor interprets this as a form of entailment between the claim and rationale.
• Figure 2: An overview of the proposed approach during training (bold in blue) and inference (bold in red). The NLI predictor only has access to the task label during training. The NLI predictor is initially fine-tuned using a limited set of annotated rationales, before generating artificial targets for the explainer. Cross-checking alignment is conducted during inference against the predictor (explained in Section \ref{['sec:inference']})
• Figure 3: Task and Plausibility performance drop when there is no further fine-tuning on the NLI predictor (10% data). The metrics are computed similarly to robustness using ( \ref{['eq:4']}) and ( \ref{['eq:5']}) and are presented in normalized percentages.
• Figure 4: Example of query and input document where the sentences highlighted in green refer to the NLI predicator without fine-tuning. Yellow refers to the annotated rationale as well as extracted by the fine-tuned predictor.