Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

How Good is ChatGPT at Audiovisual Deepfake Detection: A Comparative Study of ChatGPT, AI Models and Human Perception

Sahibzada Adil Shahzad, Ammarah Hashmi, Yan-Tsung Peng, Yu Tsao, Hsin-Min Wang

TL;DR

This study examines the detection capabilities of a large language model (LLM) (i.e., ChatGPT) to identify and account for any possible visual and auditory artifacts and manipulations in audiovisual deepfake content and discusses the limitations of ChatGPT for multimedia forensic tasks.

Abstract

Multimodal deepfakes involving audiovisual manipulations are a growing threat because they are difficult to detect with the naked eye or using unimodal deep learningbased forgery detection methods. Audiovisual forensic models, while more capable than unimodal models, require large training datasets and are computationally expensive for training and inference. Furthermore, these models lack interpretability and often do not generalize well to unseen manipulations. In this study, we examine the detection capabilities of a large language model (LLM) (i.e., ChatGPT) to identify and account for any possible visual and auditory artifacts and manipulations in audiovisual deepfake content. Extensive experiments are conducted on videos from a benchmark multimodal deepfake dataset to evaluate the detection performance of ChatGPT and compare it with the detection capabilities of state-of-the-art multimodal forensic models and humans. Experimental results demonstrate the importance of domain knowledge and prompt engineering for video forgery detection tasks using LLMs. Unlike approaches based on end-to-end learning, ChatGPT can account for spatial and spatiotemporal artifacts and inconsistencies that may exist within or across modalities. Additionally, we discuss the limitations of ChatGPT for multimedia forensic tasks.

How Good is ChatGPT at Audiovisual Deepfake Detection: A Comparative Study of ChatGPT, AI Models and Human Perception

TL;DR

This study examines the detection capabilities of a large language model (LLM) (i.e., ChatGPT) to identify and account for any possible visual and auditory artifacts and manipulations in audiovisual deepfake content and discusses the limitations of ChatGPT for multimedia forensic tasks.

Abstract

Multimodal deepfakes involving audiovisual manipulations are a growing threat because they are difficult to detect with the naked eye or using unimodal deep learningbased forgery detection methods. Audiovisual forensic models, while more capable than unimodal models, require large training datasets and are computationally expensive for training and inference. Furthermore, these models lack interpretability and often do not generalize well to unseen manipulations. In this study, we examine the detection capabilities of a large language model (LLM) (i.e., ChatGPT) to identify and account for any possible visual and auditory artifacts and manipulations in audiovisual deepfake content. Extensive experiments are conducted on videos from a benchmark multimodal deepfake dataset to evaluate the detection performance of ChatGPT and compare it with the detection capabilities of state-of-the-art multimodal forensic models and humans. Experimental results demonstrate the importance of domain knowledge and prompt engineering for video forgery detection tasks using LLMs. Unlike approaches based on end-to-end learning, ChatGPT can account for spatial and spatiotemporal artifacts and inconsistencies that may exist within or across modalities. Additionally, we discuss the limitations of ChatGPT for multimedia forensic tasks.

Paper Structure

This paper contains 19 sections, 6 equations, 2 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Methodology
  4. Prompt Design
  5. Input of LLM Model
  6. Audiovisual Analysis
  7. Prediction Assignment
  8. ChatGPT vs Human vs AI Models
  9. Experiments and Results
  10. Dataset Selection
  11. Evaluation Metrics
  12. Results
  13. Comparing Different Text Prompts
  14. Comparing ChatGPT with Human and AI models
  15. Ablation study
  16. ...and 4 more sections

Figures (2)

  • Figure 1: Illustration of audiovisual deepfake manipulations. Original video content is represented as RVRA (real video with real audio. Through deepfake manipulation techniques, three manipulated types are generated: FVRA (fake video with real audio), RVFA (real video with fake audio), and FVFA (fake video with fake audio). Blue text represents the "real modality" of the video content, while red text represents the "fake modality".
  • Figure 2: Illustration of the multimodal capabilities of a large language model, which takes inputs such as visual frames, audio speech, and text prompts to produce outputs that include audiovisual analysis, interpretation, and authenticity prediction.