Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Tell Me Why: Explainable Public Health Fact-Checking with Large Language Models

Majid Zarharan, Pascal Wullschleger, Babak Behkam Kia, Mohammad Taher Pilehvar, Jennifer Foster

TL;DR

The paper tackles explainable fact-checking in public health by evaluating a broad set of LLMs across veracity prediction, explanation generation, and joint tasks using the PUBHEALTH dataset. It compares zero-shot, few-shot, and parameter-efficient fine-tuning (PEFT) regimes for closed- and open-source models, employing both automatic metrics and novel human-evaluation guidelines. Key findings show GPT-4 excels in zero-shot explanations, while PEFT enables open-source models to rival or surpass GPT-4 in few-shot or trained settings, with human judges highlighting nuanced gaps in gold explanations. The work provides new evaluation guidelines, demonstrates practical insights for deploying explainable health fact-checking systems, and discusses limitations and avenues for future research.

Abstract

This paper presents a comprehensive analysis of explainable fact-checking through a series of experiments, focusing on the ability of large language models to verify public health claims and provide explanations or justifications for their veracity assessments. We examine the effectiveness of zero/few-shot prompting and parameter-efficient fine-tuning across various open and closed-source models, examining their performance in both isolated and joint tasks of veracity prediction and explanation generation. Importantly, we employ a dual evaluation approach comprising previously established automatic metrics and a novel set of criteria through human evaluation. Our automatic evaluation indicates that, within the zero-shot scenario, GPT-4 emerges as the standout performer, but in few-shot and parameter-efficient fine-tuning contexts, open-source models demonstrate their capacity to not only bridge the performance gap but, in some instances, surpass GPT-4. Human evaluation reveals yet more nuance as well as indicating potential problems with the gold explanations.

Tell Me Why: Explainable Public Health Fact-Checking with Large Language Models

TL;DR

The paper tackles explainable fact-checking in public health by evaluating a broad set of LLMs across veracity prediction, explanation generation, and joint tasks using the PUBHEALTH dataset. It compares zero-shot, few-shot, and parameter-efficient fine-tuning (PEFT) regimes for closed- and open-source models, employing both automatic metrics and novel human-evaluation guidelines. Key findings show GPT-4 excels in zero-shot explanations, while PEFT enables open-source models to rival or surpass GPT-4 in few-shot or trained settings, with human judges highlighting nuanced gaps in gold explanations. The work provides new evaluation guidelines, demonstrates practical insights for deploying explainable health fact-checking systems, and discusses limitations and avenues for future research.

Abstract

This paper presents a comprehensive analysis of explainable fact-checking through a series of experiments, focusing on the ability of large language models to verify public health claims and provide explanations or justifications for their veracity assessments. We examine the effectiveness of zero/few-shot prompting and parameter-efficient fine-tuning across various open and closed-source models, examining their performance in both isolated and joint tasks of veracity prediction and explanation generation. Importantly, we employ a dual evaluation approach comprising previously established automatic metrics and a novel set of criteria through human evaluation. Our automatic evaluation indicates that, within the zero-shot scenario, GPT-4 emerges as the standout performer, but in few-shot and parameter-efficient fine-tuning contexts, open-source models demonstrate their capacity to not only bridge the performance gap but, in some instances, surpass GPT-4. Human evaluation reveals yet more nuance as well as indicating potential problems with the gold explanations.
Paper Structure (38 sections, 4 figures, 18 tables)

This paper contains 38 sections, 4 figures, 18 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Fact Checking Datasets.
  4. Methods.
  5. Methodology
  6. Experimental Details
  7. Selected LLMs
  8. Dataset
  9. Context Summarization
  10. Evaluation
  11. Automatic Evaluation
  12. Strong Global Coherence (SGC).
  13. Weak Global Coherence (WGC).
  14. Local Coherence (LC).
  15. Human Evaluation
  16. ...and 23 more sections

Figures (4)

  • Figure 1: The pipeline for veracity prediction, explanation generation, and the joint setting.
  • Figure 2: The number of words in the main text in different sets of PUBHEALTH dataset
  • Figure 3: A screenshot of our annotation tool.
  • Figure 4: The correlation between different human evaluation metrics for each task.