Is Seeing Believing? Evaluating Human Sensitivity to Synthetic Video

Abstract

Advances in machine learning have enabled the creation of realistic synthetic videos known as deepfakes. As deepfakes proliferate, concerns about rapid spread of disinformation and manipulation of public perception are mounting. Despite the alarming implications, our understanding of how individuals perceive synthetic media remains limited, obstructing the development of effective mitigation strategies. This paper aims to narrow this gap by investigating human responses to visual and auditory distortions of videos and deepfake-generated visuals and narration. In two between-subjects experiments, we study whether audio-visual distortions affect cognitive processing, such as subjective credibility assessment and objective learning outcomes. A third study reveals that artifacts from deepfakes influence credibility. The three studies show that video distortions and deepfake artifacts can reduce credibility. Our research contributes to the ongoing exploration of the cognitive processes involved in the evaluation and perception of synthetic videos, and underscores the need for further theory development concerning deepfake exposure.

Figures (25)

• Figure 1: Stills of the two stimulus videos used in Study I illustrating the differences between the experimental conditions: (a) a still from the unaltered, recorded video shown to participants in the baseline condition, and (b) a still from the distorted video shown to participants in the reduced visual clarity condition
• Figure 2: Stills showing the stimulus videos used in Study III illustrating the visual differences between the experimental conditions: (a) a still from the unaltered, recorded video shown to participants in the conditions baseline and synthesized narration, and (b) a still from the deepfake video shown to participants in the synthesized video and synthesized video and narration conditions.
• Figure 3: Measurements of Message Credibility (a) and Processing Fluency (b) in Study I divided by experimental condition. Dots represent mean values, horizontal lines indicate bootstrapped confidence intervals. Significant differences are indicated with brackets and asterisks: One asterisk (*) $p < 0.05$, two asterisks (**) $p < 0.01$, three asterisks (***) $p < 0.0001$.
• Figure 4: Percentage of responses to the alteration detection question by treatment group in Study I.
• Figure 5: Measurements of Message Credibility (a), Processing Fluency (b), Factual and conceptual learning (c), and Source Vividness (d) in Study II divided by experimental condition. Dots represent mean values, horizontal lines indicate bootstrapped confidence intervals. Significant differences according to Bonferroni-corrected tests are indicated with brackets and asterisks: One asterisk (*) $p < 0.05$, two asterisks (**) $p < 0.01$.