Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

AfrIFact: Cultural Information Retrieval, Evidence Extraction and Fact Checking for African Languages

Israel Abebe Azime, Jesujoba Oluwadara Alabi, Crystina Zhang, Iffat Maab, Atnafu Lambebo Tonja, Tadesse Destaw Belay, Folasade Peace Alabi, Salomey Osei, Saminu Mohammad Aliyu, Nkechinyere Faith Aguobi, Bontu Fufa Balcha, Blessing Kudzaishe Sibanda, Davis David, Mouhamadane Mboup, Daud Abolade, Neo Putini, Philipp Slusallek, David Ifeoluwa Adelani, Dietrich Klakow

Abstract

Assessing the veracity of a claim made online is a complex and important task with real-world implications. When these claims are directed at communities with limited access to information and the content concerns issues such as healthcare and culture, the consequences intensify, especially in low-resource languages. In this work, we introduce AfrIFact, a dataset that covers the necessary steps for automatic fact-checking (i.e., information retrieval, evidence extraction, and fact checking), in ten African languages and English. Our evaluation results show that even the best embedding models lack cross-lingual retrieval capabilities, and that cultural and news documents are easier to retrieve than healthcare-domain documents, both in large corpora and in single documents. We show that LLMs lack robust multilingual fact-verification capabilities in African languages, while few-shot prompting improves performance by up to 43% in AfriqueQwen-14B, and task-specific fine-tuning further improves fact-checking accuracy by up to 26%. These findings, along with our release of the AfrIFact dataset, encourage work on low-resource information retrieval, evidence retrieval, and fact checking.

AfrIFact: Cultural Information Retrieval, Evidence Extraction and Fact Checking for African Languages

Abstract

Assessing the veracity of a claim made online is a complex and important task with real-world implications. When these claims are directed at communities with limited access to information and the content concerns issues such as healthcare and culture, the consequences intensify, especially in low-resource languages. In this work, we introduce AfrIFact, a dataset that covers the necessary steps for automatic fact-checking (i.e., information retrieval, evidence extraction, and fact checking), in ten African languages and English. Our evaluation results show that even the best embedding models lack cross-lingual retrieval capabilities, and that cultural and news documents are easier to retrieve than healthcare-domain documents, both in large corpora and in single documents. We show that LLMs lack robust multilingual fact-verification capabilities in African languages, while few-shot prompting improves performance by up to 43% in AfriqueQwen-14B, and task-specific fine-tuning further improves fact-checking accuracy by up to 26%. These findings, along with our release of the AfrIFact dataset, encourage work on low-resource information retrieval, evidence retrieval, and fact checking.

Paper Structure

This paper contains 36 sections, 9 figures, 15 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Automated Fact Checking (AFC)
  4. Information Retrieval (IR)
  5. AfrIFact Dataset
  6. AfrIFact-Health
  7. AfrIFact-Culture-News
  8. Agreement Scores
  9. Methodology
  10. Information Retrieval Documents:
  11. Information Retrieval & Evidence Extraction Evaluation:
  12. Claim Labeling Evaluation:
  13. Language Models:
  14. Few-shot and Fine-tuning Settings:
  15. Results and Analysis
  16. ...and 21 more sections

Figures (9)

  • Figure 1: Three steps for verifying factual claims are covered by our dataset. (A) information retrieval to find documents related to a claim, (B) extraction of sentence-level evidence supporting or refuting the claim, and (C) fact-checking using the claim, retrieved document, and extracted evidence with LLMs.
  • Figure 2: Illustration of the data construction process for AfrIFact-Health on shared health-data approach for lack of structured native health documents and AfrIFact-Culture-News culturally grounded, natively sourced data approach
  • Figure 3: nDCG@10 scores on the Health domain when retrieving from different corpora and evaluated on relevant documents in mono- or multi-languages. "A. Avg" shows the average score on African languages. Scores with label "mono" indicate that we only consider relevant documents from the same language as the query during evaluation.
  • Figure 4: Average accuracy scores on African languages of different language models on the AfrIFact fact-checking task under zero-shot and three-shot settings, with and without evidence.
  • Figure 5: Why is evidence not helping improve accuracy? Evidence introduces a conservative shift in model predictions: in Health, it reduces hallucinated SUPPORTS but increases NOT_ENOUGH_INFORMATION predictions, while in Culture, it significantly improves NEI detection by reducing false Supports classifications. In the Health domain NOT_ENOUGH_INFORMATION and SUPPORTS improve while in the culture domain, the model tends to be good at identifying NOT_ENOUGH_INFORMATION when evidence is provided. In both domains, the model confusion between SUPPORTS and REFUTES improves with evidence provided.
  • ...and 4 more figures