Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

ViFactCheck: A New Benchmark Dataset and Methods for Multi-domain News Fact-Checking in Vietnamese

Tran Thai Hoa, Tran Quang Duy, Khanh Quoc Tran, Kiet Van Nguyen

TL;DR

ViFactCheck addresses the gap in Vietnamese fact-checking by introducing the first public, multi-domain benchmark with 7,232 human-annotated claim–evidence pairs across 12 topics and a high inter-annotator agreement (0.83 Fleiss Kappa). The authors evaluate a spectrum of baselines, including PLMs and open-source LLMs, using fine-tuning and prompting, and report Gemma achieving a macro F1 of 89.90% on evidence-based verification tasks. They analyze retrieval strategies, multi-evidence reasoning, and error modes, showing that evidence retrieval and model scale strongly influence performance, with Gold Evidence generally yielding higher accuracy than Full Context. The work releases datasets, model checkpoints, pipelines, and code on GitHub, aiming to spur progress in fact-checking for low-resource languages and enabling broader research and deployment.

Abstract

The rapid spread of information in the digital age highlights the critical need for effective fact-checking tools, particularly for languages with limited resources, such as Vietnamese. In response to this challenge, we introduce ViFactCheck, the first publicly available benchmark dataset designed specifically for Vietnamese fact-checking across multiple online news domains. This dataset contains 7,232 human-annotated pairs of claim-evidence combinations sourced from reputable Vietnamese online news, covering 12 diverse topics. It has been subjected to a meticulous annotation process to ensure high quality and reliability, achieving a Fleiss Kappa inter-annotator agreement score of 0.83. Our evaluation leverages state-of-the-art pre-trained and large language models, employing fine-tuning and prompting techniques to assess performance. Notably, the Gemma model demonstrated superior effectiveness, with an impressive macro F1 score of 89.90%, thereby establishing a new standard for fact-checking benchmarks. This result highlights the robust capabilities of Gemma in accurately identifying and verifying facts in Vietnamese. To further promote advances in fact-checking technology and improve the reliability of digital media, we have made the ViFactCheck dataset, model checkpoints, fact-checking pipelines, and source code freely available on GitHub. This initiative aims to inspire further research and enhance the accuracy of information in low-resource languages.

ViFactCheck: A New Benchmark Dataset and Methods for Multi-domain News Fact-Checking in Vietnamese

TL;DR

ViFactCheck addresses the gap in Vietnamese fact-checking by introducing the first public, multi-domain benchmark with 7,232 human-annotated claim–evidence pairs across 12 topics and a high inter-annotator agreement (0.83 Fleiss Kappa). The authors evaluate a spectrum of baselines, including PLMs and open-source LLMs, using fine-tuning and prompting, and report Gemma achieving a macro F1 of 89.90% on evidence-based verification tasks. They analyze retrieval strategies, multi-evidence reasoning, and error modes, showing that evidence retrieval and model scale strongly influence performance, with Gold Evidence generally yielding higher accuracy than Full Context. The work releases datasets, model checkpoints, pipelines, and code on GitHub, aiming to spur progress in fact-checking for low-resource languages and enabling broader research and deployment.

Abstract

The rapid spread of information in the digital age highlights the critical need for effective fact-checking tools, particularly for languages with limited resources, such as Vietnamese. In response to this challenge, we introduce ViFactCheck, the first publicly available benchmark dataset designed specifically for Vietnamese fact-checking across multiple online news domains. This dataset contains 7,232 human-annotated pairs of claim-evidence combinations sourced from reputable Vietnamese online news, covering 12 diverse topics. It has been subjected to a meticulous annotation process to ensure high quality and reliability, achieving a Fleiss Kappa inter-annotator agreement score of 0.83. Our evaluation leverages state-of-the-art pre-trained and large language models, employing fine-tuning and prompting techniques to assess performance. Notably, the Gemma model demonstrated superior effectiveness, with an impressive macro F1 score of 89.90%, thereby establishing a new standard for fact-checking benchmarks. This result highlights the robust capabilities of Gemma in accurately identifying and verifying facts in Vietnamese. To further promote advances in fact-checking technology and improve the reliability of digital media, we have made the ViFactCheck dataset, model checkpoints, fact-checking pipelines, and source code freely available on GitHub. This initiative aims to inspire further research and enhance the accuracy of information in low-resource languages.

Paper Structure

This paper contains 49 sections, 13 figures, 10 tables.

Table of Contents

  1. Introduction
  2. Fundamental of Fact-Checking
  3. Foundational Benchmark Datasets
  4. Advanced Methods in Fact-Checking
  5. Vietnamese Research on Fact-Checking
  6. Dataset Creation Process
  7. Data Collection
  8. Dataset Annotation
  9. Pilot Annotation
  10. Main Annotation
  11. Validation of Annotation
  12. Words overlap and Semantic similarity analysis
  13. Experiment and Results
  14. Baseline models
  15. Software and Hardware Configurations
  16. ...and 34 more sections

Figures (13)

  • Figure 1: An example of the Vietnamese fact-checking task. Words highlighted in blue represent key evidence used to support the classification of the claim as "Supported".
  • Figure 2: The ViFactCheck dataset contruction process.
  • Figure 3: Comparison of the labeling pipelines in the FEVER and ViFactCheck datasets.
  • Figure 4: Comparative performance of various text retrieval models across different Top-K settings.
  • Figure 5: Distributions of errors.
  • ...and 8 more figures