Seeing, Hearing, and Knowing Together: Multimodal Strategies in Deepfake Videos Detection

TL;DR

The paper investigates how humans detect real versus deepfake videos by examining visual, audio, and knowledge-based cues. Using a controlled experiment with 195 participants evaluating 20 videos and reporting cue usage, the authors quantify accuracy and calibration (ECE) and apply association-rule mining to reveal cue combinations, visualizing interaction networks. Findings show real videos benefit from multimodal cue integration, achieving high accuracy and good calibration, while deepfakes remain challenging and often yield poorer calibration, with limited gains from adding more strategies. The work advances understanding of human detection strategies and offers design guidance for media literacy interventions that foster effective, calibrated cue use in everyday media consumption. It also highlights implications for human-AI collaboration and policy measures to counter synthetic media threats.

Abstract

As deepfake videos become increasingly difficult for people to recognise, understanding the strategies humans use is key to designing effective media literacy interventions. We conducted a study with 195 participants between the ages of 21 and 40, who judged real and deepfake videos, rated their confidence, and reported the cues they relied on across visual, audio, and knowledge strategies. Participants were more accurate with real videos than with deepfakes and showed lower expected calibration error for real content. Through association rule mining, we identified cue combinations that shaped performance. Visual appearance, vocal, and intuition often co-occurred for successful identifications, which highlights the importance of multimodal approaches in human detection. Our findings show which cues help or hinder detection and suggest directions for designing media literacy tools that guide effective cue use. Building on these insights can help people improve their identification skills and become more resilient to deceptive digital media.

Figures (9)

• Figure 1: Confusion Matrix.
• Figure 2: Top 10 accuracy by strategy cue combinations for deepfake vs. real videos. The chart compares how often specific multiple cue sets led to detection performance. Bars show mean accuracy and ECE for each cue combination, separately for real and deepfake videos.
• Figure 3: Network visualisation of strategy cue accuracy for deepfake videos and real videos. Node size represents cue frequency of use. Edge thickness represents average accuracy when cues were combined. Colours indicate modality (blue = visual, teal = audio, orange = knowledge). Node labels display cue names (app-appearance, env-environment, prodqual-production quality; soundqual-sound quality; sources-information sources, know-prior knowledge of person/video content/ video context). This figure highlights which cue combinations supported successful detection for each video type.
• Figure 4: Network visualisation of strategy cue calibration (ECE-inverted) for deepfake videos and real videos. Node size represents cue frequency of use. Edge thickness represents better alignment between confidence and accuracy (lower ECE). Colours indicate modality (blue = visual, teal = audio, orange = knowledge). This figure illustrates which cues contributed to more calibrated judgments across video types.
• Figure 5: Deepfake video of Jeremy Corbyn