Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Collab: Fostering Critical Identification of Deepfake Videos on Social Media via Synergistic Annotation

Shuning Zhang, Linzhi Wang, Shixuan Li, Yuanyuan Wu, Yuwei Chuai, Luoxi Chen, Xin Yi, Hewu Li

TL;DR

Collab introduces a CI/SI-informed framework for collaborative deepfake video identification, integrating a spatio-temporal labeling interface, a confidence-weighted 3D IoU-based aggregation, and a hierarchical demonstration to guide scrutiny without provoking conformity bias. In a 7-day online study with 90 participants, Collab achieved an F1-score of $0.883$, outperforming baselines lacking aggregation or demonstration and enhancing reflective judgment. The approach demonstrates that structured crowd input, when properly aggregated and presented, can match or exceed automated detectors in localization accuracy while fostering critical evaluation. These findings support scalable, user-centered moderation on social media and offer design principles for integrating crowd intelligence with expert oversight in multimedia misinformation contexts.

Abstract

Identifying deepfake videos on social media platforms is challenged by dynamic spatio-temporal artifacts and inadequate user tools. This hinders both critical viewing by users and scalable moderation on platforms. Here, we present Collab, a web plugin enabling users to collaboratively annotate deepfake videos. Collab integrates three key components: (i) an intuitive interface for spatio-temporal labeling where users provide confidence scores and rationales, facilitating detailed input even from non-experts, (ii) a novel confidence-weighted spatio-temporal Intersection-over-Union (IoU) algorithm to aggregate diverse user annotations into accurate aggregations, and (iii) a hierarchical demonstration strategy presenting aggregated results to guide attention toward contentious regions and foster critical evaluation. A seven-day online study (N=90), where participants annotated suspicious videos when viewing an online experimental platforms, compared Collab against two conditions without aggregation or demonstration respectively. Collab significantly improved identification accuracy and enhanced reflection compared to non-demonstration condition, while outperforming non-aggregation condition for its novelty and effectiveness.

Collab: Fostering Critical Identification of Deepfake Videos on Social Media via Synergistic Annotation

TL;DR

Collab introduces a CI/SI-informed framework for collaborative deepfake video identification, integrating a spatio-temporal labeling interface, a confidence-weighted 3D IoU-based aggregation, and a hierarchical demonstration to guide scrutiny without provoking conformity bias. In a 7-day online study with 90 participants, Collab achieved an F1-score of , outperforming baselines lacking aggregation or demonstration and enhancing reflective judgment. The approach demonstrates that structured crowd input, when properly aggregated and presented, can match or exceed automated detectors in localization accuracy while fostering critical evaluation. These findings support scalable, user-centered moderation on social media and offer design principles for integrating crowd intelligence with expert oversight in multimedia misinformation contexts.

Abstract

Identifying deepfake videos on social media platforms is challenged by dynamic spatio-temporal artifacts and inadequate user tools. This hinders both critical viewing by users and scalable moderation on platforms. Here, we present Collab, a web plugin enabling users to collaboratively annotate deepfake videos. Collab integrates three key components: (i) an intuitive interface for spatio-temporal labeling where users provide confidence scores and rationales, facilitating detailed input even from non-experts, (ii) a novel confidence-weighted spatio-temporal Intersection-over-Union (IoU) algorithm to aggregate diverse user annotations into accurate aggregations, and (iii) a hierarchical demonstration strategy presenting aggregated results to guide attention toward contentious regions and foster critical evaluation. A seven-day online study (N=90), where participants annotated suspicious videos when viewing an online experimental platforms, compared Collab against two conditions without aggregation or demonstration respectively. Collab significantly improved identification accuracy and enhanced reflection compared to non-demonstration condition, while outperforming non-aggregation condition for its novelty and effectiveness.
Paper Structure (46 sections, 9 figures, 1 table, 1 algorithm)

This paper contains 46 sections, 9 figures, 1 table, 1 algorithm.

Table of Contents

  1. Introduction
  2. Background & Related Work
  3. Deepfake Detection For Machines and Humans
  4. Sociotechnical Dynamics and Human-Centered Approaches
  5. User Interface Design for Collaborative Annotating
  6. Design and Implementation of Collab
  7. Design Guidelines
  8. Overall Design Goals
  9. Design Guidelines for Each Process
  10. Labeling
  11. Aggregation
  12. Demonstration
  13. System Design
  14. Annotation Design
  15. Annotation Interface and Workflow
  16. ...and 31 more sections

Figures (9)

  • Figure 1: The design dimensions of Collab.
  • Figure 2: The design candidates of , where the choices selected by are highlighted with green marks.
  • Figure 3: Candidate artifact demonstration designs. (a) E-reader Marker-based representation, (b) Hybrid e-reader marker/region-based representation, and (c) Region-based bounding box representation. Details are presented upon selection. The ① represents a marker.
  • Figure 4: The interface illustration for different techniques, where No Agg shows non-aggregated labels, and No Label shows no labels.
  • Figure 5: Average accuracy of different techniques across seven days.
  • ...and 4 more figures