Human-AI Ensembles Improve Deepfake Detection in Low-to-Medium Quality Videos

Abstract

Deepfake detection is widely framed as a machine learning problem, yet how humans and AI detectors compare under realistic conditions remains poorly understood. We evaluate 200 human participants and 95 state-of-the-art AI detectors across two datasets: DF40, a standard benchmark, and CharadesDF, a novel dataset of videos of everyday activities. CharadesDF was recorded using mobile phones leading to low/moderate quality videos compared to the more professionally captured DF40. Humans outperform AI detectors on both datasets, with the gap widening in the case of CharadesDF where AI accuracy collapses to near chance (0.537) while humans maintain robust performance (0.784). Human and AI errors are complementary: humans miss high-quality deepfakes while AI detectors flag authentic videos as fake, and hybrid human-AI ensembles reduce high-confidence errors. These findings suggest that effective real-world deepfake detection, especially in non-professionally produced videos, requires human-AI collaboration rather than AI algorithms alone.

Figures (10)

• Figure 1: Examples of video recordings from the CharadesDF dataset. Participants recorded videos in their home environments while performing everyday activities such as drinking from cups, holding objects, stretching, and working at desks. These conditions produce substantial variability in lighting, camera angles, face visibility, and image quality—challenges common in authentic user-generated content but underrepresented in existing deepfake detection benchmarks.
• Figure 2: Distribution of participant-level accuracy for human participants and AI detectors. (A) DF40 dataset. (B) CharadesDF dataset. Each data point represents one participant's overall accuracy across all videos they evaluated.
• Figure 3: Complementary failure patterns between human and AI ensembles. Each point represents a video, plotted by human ensemble probability (x-axis) versus AI ensemble probability (y-axis) of being a deepfake. Dashed lines indicate the decision threshold (0.5). Small circles show videos without catastrophic failures (teal: real videos; orange: deepfakes). Squares indicate videos where only the human ensemble failed catastrophically; triangles indicate videos where only the AI ensemble failed. No videos caused catastrophic failures in both groups simultaneously (no X markers). (A) DF40 dataset. (B) CharadesDF dataset.
• Figure 4: Catastrophic failure rates across threshold definitions. CFR was computed for threshold values ranging from 0.5 (any probability error $\ge 0.5$ is catastrophic) to 1.0 (only complete misclassification with probability error = 1.0). Dashed lines show mean individual performance; solid lines show ensemble performance. Quality-weighted voting was used for all ensembles. (A) DF40 dataset. (B) CharadesDF dataset.
• Figure 5: Detection accuracy as a function of image quality factors. Smoothed performance curves showing the relationship between quality features and detection accuracy for human individuals (teal, dashed), AI detectors (orange, dashed), human ensembles (teal, solid), AI ensembles (orange, solid), and hybrid ensembles (purple, solid). Shaded regions indicate 95% confidence intervals. We indicate correlations ($\rho$) in each subplot for both humans ($H$) and AI detectors ($M$). Rows correspond to DF40 (top) and CharadesDF (bottom) datasets. (A) Bounding box area ratio. (B) Inter-ocular distance (IOD). (C) Signal-to-noise ratio (SNR). (D) Contrast. (E) Color balance.