Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

FactFinders at CheckThat! 2024: Refining Check-worthy Statement Detection with LLMs through Data Pruning

Yufeng Li, Rrubaa Panchendrarajan, Arkaitz Zubiaga

TL;DR

This study investigates the application of eight prominent open-source LLMs with fine-tuning and prompt engineering to identify check-worthy statements from political transcriptions and proposes a two-step data pruning approach to automatically identify high-quality training data instances for effective learning.

Abstract

The rapid dissemination of information through social media and the Internet has posed a significant challenge for fact-checking, among others in identifying check-worthy claims that fact-checkers should pay attention to, i.e. filtering claims needing fact-checking from a large pool of sentences. This challenge has stressed the need to focus on determining the priority of claims, specifically which claims are worth to be fact-checked. Despite advancements in this area in recent years, the application of large language models (LLMs), such as GPT, has only recently drawn attention in studies. However, many open-source LLMs remain underexplored. Therefore, this study investigates the application of eight prominent open-source LLMs with fine-tuning and prompt engineering to identify check-worthy statements from political transcriptions. Further, we propose a two-step data pruning approach to automatically identify high-quality training data instances for effective learning. The efficiency of our approach is demonstrated through evaluations on the English language dataset as part of the check-worthiness estimation task of CheckThat! 2024. Further, the experiments conducted with data pruning demonstrate that competitive performance can be achieved with only about 44\% of the training data. Our team ranked first in the check-worthiness estimation task in the English language.

FactFinders at CheckThat! 2024: Refining Check-worthy Statement Detection with LLMs through Data Pruning

TL;DR

This study investigates the application of eight prominent open-source LLMs with fine-tuning and prompt engineering to identify check-worthy statements from political transcriptions and proposes a two-step data pruning approach to automatically identify high-quality training data instances for effective learning.

Abstract

The rapid dissemination of information through social media and the Internet has posed a significant challenge for fact-checking, among others in identifying check-worthy claims that fact-checkers should pay attention to, i.e. filtering claims needing fact-checking from a large pool of sentences. This challenge has stressed the need to focus on determining the priority of claims, specifically which claims are worth to be fact-checked. Despite advancements in this area in recent years, the application of large language models (LLMs), such as GPT, has only recently drawn attention in studies. However, many open-source LLMs remain underexplored. Therefore, this study investigates the application of eight prominent open-source LLMs with fine-tuning and prompt engineering to identify check-worthy statements from political transcriptions. Further, we propose a two-step data pruning approach to automatically identify high-quality training data instances for effective learning. The efficiency of our approach is demonstrated through evaluations on the English language dataset as part of the check-worthiness estimation task of CheckThat! 2024. Further, the experiments conducted with data pruning demonstrate that competitive performance can be achieved with only about 44\% of the training data. Our team ranked first in the check-worthiness estimation task in the English language.
Paper Structure (18 sections, 6 figures, 9 tables)

This paper contains 18 sections, 6 figures, 9 tables.

Table of Contents

  1. Introduction
  2. Methodology
  3. Dataset
  4. LLMs for Check-worthy Statement Detection
  5. Large Language Models
  6. Prompt Engineering
  7. Effective Fine-tuning
  8. Data Pruning for Effective Learning
  9. Step 1 - Identifying Informative Sentences
  10. Step 2 - Under Sampling using Condensed Nearest Neighbour
  11. Results
  12. Hyper-parameters and Environment Setting
  13. Evaluation Metrics
  14. Comparison of LLMs
  15. Consistency Analysis
  16. ...and 3 more sections

Figures (6)

  • Figure 1: Distribution of Text Length in each Partition
  • Figure 2: Word cloud indicating verb types and their frequencies in the training data.
  • Figure 3: Distribution of verb types in the training data
  • Figure 4: 2D visualization of the Training Data.
  • Figure 5: Consistency of Llama2-7b with Iterations
  • ...and 1 more figures