Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

ViClaim: A Multilingual Multilabel Dataset for Automatic Claim Detection in Videos

Patrick Giedemann, Pius von Däniken, Jan Deriu, Alvaro Rodrigo, Anselmo Peñas, Mark Cieliebak

TL;DR

ViClaim addresses the gap in multilingual, video-based misinformation detection by presenting a sentence-level, multi-label dataset of 1,798 YouTube Shorts across English, German, and Spanish, labeled for FCW, FNC, and OPN across six topics. A custom annotation tool, multi-annotator workflow, and MACE-based soft-labeling yield robust yet imperfect annotations, enabling strong baseline results (e.g., up to $F_1$ of $0.899$ for FCW in cross-validation) while highlighting generalization challenges to unseen topics. Baseline experiments fine-tune four multilingual models with QLoRA, demonstrating solid in-domain performance but notable domain transfer gaps, especially for non-political domains like League of Legends. By releasing the dataset, tooling, and baselines, the work sets a foundation for advancing multimodal, multilingual misinformation detection in video transcripts and motivates future work on multimedia integration and cross-domain adaptation.

Abstract

The growing influence of video content as a medium for communication and misinformation underscores the urgent need for effective tools to analyze claims in multilingual and multi-topic settings. Existing efforts in misinformation detection largely focus on written text, leaving a significant gap in addressing the complexity of spoken text in video transcripts. We introduce ViClaim, a dataset of 1,798 annotated video transcripts across three languages (English, German, Spanish) and six topics. Each sentence in the transcripts is labeled with three claim-related categories: fact-check-worthy, fact-non-check-worthy, or opinion. We developed a custom annotation tool to facilitate the highly complex annotation process. Experiments with state-of-the-art multilingual language models demonstrate strong performance in cross-validation (macro F1 up to 0.896) but reveal challenges in generalization to unseen topics, particularly for distinct domains. Our findings highlight the complexity of claim detection in video transcripts. ViClaim offers a robust foundation for advancing misinformation detection in video-based communication, addressing a critical gap in multimodal analysis.

ViClaim: A Multilingual Multilabel Dataset for Automatic Claim Detection in Videos

TL;DR

ViClaim addresses the gap in multilingual, video-based misinformation detection by presenting a sentence-level, multi-label dataset of 1,798 YouTube Shorts across English, German, and Spanish, labeled for FCW, FNC, and OPN across six topics. A custom annotation tool, multi-annotator workflow, and MACE-based soft-labeling yield robust yet imperfect annotations, enabling strong baseline results (e.g., up to of for FCW in cross-validation) while highlighting generalization challenges to unseen topics. Baseline experiments fine-tune four multilingual models with QLoRA, demonstrating solid in-domain performance but notable domain transfer gaps, especially for non-political domains like League of Legends. By releasing the dataset, tooling, and baselines, the work sets a foundation for advancing multimodal, multilingual misinformation detection in video transcripts and motivates future work on multimedia integration and cross-domain adaptation.

Abstract

The growing influence of video content as a medium for communication and misinformation underscores the urgent need for effective tools to analyze claims in multilingual and multi-topic settings. Existing efforts in misinformation detection largely focus on written text, leaving a significant gap in addressing the complexity of spoken text in video transcripts. We introduce ViClaim, a dataset of 1,798 annotated video transcripts across three languages (English, German, Spanish) and six topics. Each sentence in the transcripts is labeled with three claim-related categories: fact-check-worthy, fact-non-check-worthy, or opinion. We developed a custom annotation tool to facilitate the highly complex annotation process. Experiments with state-of-the-art multilingual language models demonstrate strong performance in cross-validation (macro F1 up to 0.896) but reveal challenges in generalization to unseen topics, particularly for distinct domains. Our findings highlight the complexity of claim detection in video transcripts. ViClaim offers a robust foundation for advancing misinformation detection in video-based communication, addressing a critical gap in multimodal analysis.

Paper Structure

This paper contains 37 sections, 5 figures, 7 tables.

Table of Contents

  1. Introduction
  2. Release.
  3. Related Work
  4. Data Collection
  5. Annotation Tool
  6. Tag-Set Description
  7. Topic Selection
  8. Video Selection
  9. Annotation Management
  10. Participant Training.
  11. Quality Control
  12. Annotator Agreement.
  13. Label Normalization.
  14. Data Overview
  15. Baseline Experiments
  16. ...and 22 more sections

Figures (5)

  • Figure 1: Annotation Tool. The user is shown a short video and the transcript, which is split by sentence, and then they annotate each sentence with one or more of the four tags.
  • Figure 2: For each label, each topic's appearance percentage is depicted. The labels are sorted by their overall frequency.
  • Figure 3: Form questions 1 to 12
  • Figure 4: Form questions: 13 to 24
  • Figure 5: Form questions 25 to 29