Learning to Generate and Evaluate Fact-checking Explanations with Transformers

TL;DR

Transformer-based fact-checking models that contextualise and justify their decisions by generating human-accessible explanations and models for automatic evaluation of explanations for fact-checking verdicts across different dimensions are developed.

Abstract

In an era increasingly dominated by digital platforms, the spread of misinformation poses a significant challenge, highlighting the need for solutions capable of assessing information veracity. Our research contributes to the field of Explainable Artificial Antelligence (XAI) by developing transformer-based fact-checking models that contextualise and justify their decisions by generating human-accessible explanations. Importantly, we also develop models for automatic evaluation of explanations for fact-checking verdicts across different dimensions such as \texttt{(self)-contradiction}, \texttt{hallucination}, \texttt{convincingness} and \texttt{overall quality}. By introducing human-centred evaluation methods and developing specialised datasets, we emphasise the need for aligning Artificial Intelligence (AI)-generated explanations with human judgements. This approach not only advances theoretical knowledge in XAI but also holds practical implications by enhancing the transparency, reliability and users' trust in AI-driven fact-checking systems. Furthermore, the development of our metric learning models is a first step towards potentially increasing efficiency and reducing reliance on extensive manual assessment. Based on experimental results, our best performing generative model \textsc{ROUGE-1} score of 47.77, demonstrating superior performance in generating fact-checking explanations, particularly when provided with high-quality evidence. Additionally, the best performing metric learning model showed a moderately strong correlation with human judgements on objective dimensions such as \texttt{(self)-contradiction and \texttt{hallucination}, achieving a Matthews Correlation Coefficient (MCC) of around 0.7.}

Figures (12)

• Figure 1: Our proposed methodology outlining both explanation generation and metric learning models.
• Figure 2: A visual depiction of our proposed methodology.
• Figure 3: Distribution of normalised labels for the factcheck subset.
• Figure 4: Topic model of all 14k claims in our dataset. Topics are described by the ten most frequent words associated with them. Note the prevalence of topical themes such as "Covid vaccination" or "election and government".
• Figure 5: Average agreement of workers on all tasks. Highlighted in red is the average agreement of ChatGPT.