GLCF: A Global-Local Multimodal Coherence Analysis Framework for Talking Face Generation Detection

TL;DR

This work tackles the problem of detecting talking face generation (TFG) by introducing MSTF, a large-scale, multi-scenario dataset designed to reflect real-world TFG conditions, and the Global-Local Multimodal Coherence Analysis Framework (GLCF) for detection. GLCF combines frequency-domain features, global spatiotemporal coherence, and localized multimodal fusion through three core modules: Region-Focused Smoothness Detection Module (RFSDM), Discrepancy Capture-Time Frame Aggregation Module (DCTAM), and Visual-Audio Fusion Module (V-AFM), with Local Frequency Statistic (LFS) features to robustly judge temporal and audiovisual coherence. Ablation studies show each module contributes to performance, with V-AFM providing the largest gains, and cross-dataset experiments demonstrate strong generalization beyond the MSTF domain. The dataset and framework advance practical TFG detection, enabling more robust defenses against high-fidelity talking-face forgery in diverse scenarios and modalities.

Abstract

Talking face generation (TFG) allows for producing lifelike talking videos of any character using only facial images and accompanying text. Abuse of this technology could pose significant risks to society, creating the urgent need for research into corresponding detection methods. However, research in this field has been hindered by the lack of public datasets. In this paper, we construct the first large-scale multi-scenario talking face dataset (MSTF), which contains 22 audio and video forgery techniques, filling the gap of datasets in this field. The dataset covers 11 generation scenarios and more than 20 semantic scenarios, closer to the practical application scenario of TFG. Besides, we also propose a TFG detection framework, which leverages the analysis of both global and local coherence in the multimodal content of TFG videos. Therefore, a region-focused smoothness detection module (RSFDM) and a discrepancy capture-time frame aggregation module (DCTAM) are introduced to evaluate the global temporal coherence of TFG videos, aggregating multi-grained spatial information. Additionally, a visual-audio fusion module (V-AFM) is designed to evaluate audiovisual coherence within a localized temporal perspective. Comprehensive experiments demonstrate the reasonableness and challenges of our datasets, while also indicating the superiority of our proposed method compared to the state-of-the-art deepfake detection approaches.

Figures (6)

• Figure 1: (a) Deepfake videos. (b) Talking face videos. Compared to deepfake videos, talking face videos demonstrate superior visual quality, exhibiting more realistic details, such as illumination uniformity, lip movements, teeth and so on.
• Figure 2: The construction of 11 scenarios. The six boxes in the above big image correspond to the six boxes in each small image below. The green box indicates inputs from the same source, while the red box is the opposite. Green audio denotes genuine audio, whereas red audio signifies forgery audio. Blank means no input.
• Figure 3: (a) The distribution of training set. (b) The distribution of testing set.
• Figure 4: (a) The diagram of the difference between frames. (b) The slight discrepancy in eyeball shape between frames.
• Figure 5: The overall framework of our proposed GLCF. RFSDM and DCTAM extract spatial features, interacting with audio features in V-AFM to get fusion information which is concatenated with frequency domain features extracted by the LFS.