FlashLips: 100-FPS Mask-Free Latent Lip-Sync using Reconstruction Instead of Diffusion or GANs

TL;DR

FlashLips introduces a two-stage, mask-free lip-sync framework that achieves real-time performance (>100 FPS) on a single GPU by decoupling lip control from rendering. Stage 1 uses a deterministic latent editor operating in SDXL VAE latent space to perform reconstruction-driven edits, while Stage 2 maps audio to a low-dimensional lips-pose vector via a flow-matching transformer, enabling stable, disentangled control. The approach relies on mask-free self-refinement to localize edits to the lips without explicit segmentation, and a lightweight audio-to-lips module trained with wav2vec 2.0 features to drive the editor. Empirical results show state-of-the-art lip-sync quality and perceptual fidelity with significantly faster-than-real-time speed, highlighting strong suitability for dubbing, avatars, and real-time synthesis, with mindful discussion of limitations and ethical considerations.

Abstract

We present FlashLips, a two-stage, mask-free lip-sync system that decouples lips control from rendering and achieves real-time performance running at over 100 FPS on a single GPU, while matching the visual quality of larger state-of-the-art models. Stage 1 is a compact, one-step latent-space editor that reconstructs an image using a reference identity, a masked target frame, and a low-dimensional lips-pose vector, trained purely with reconstruction losses - no GANs or diffusion. To remove explicit masks at inference, we use self-supervision: we generate mouth-altered variants of the target image, that serve as pseudo ground truth for fine-tuning, teaching the network to localize edits to the lips while preserving the rest. Stage 2 is an audio-to-pose transformer trained with a flow-matching objective to predict lips-poses vectors from speech. Together, these stages form a simple and stable pipeline that combines deterministic reconstruction with robust audio control, delivering high perceptual quality and faster-than-real-time speed.

Figures (8)

• Figure 1: Visualization of Quantitative Evaluation. Comparison of eight different lip-sync models in the cross-audio setting on seven key metrics. All results are normalized, with the best-performing model scaled to the outer edge, and the worst scaled towards the center.
• Figure 2: Overview of FlashLips. Stage 1 trains a one-step latent-space editor: first via masked reconstruction, then via a mask-free self-refinement step that learns to localize edits without segmentation. Stage 2 trains an audio-to-lips model that predicts the lips-pose vector used in Stage 1. At inference, predicted lip poses drive the LipsChange network to produce lip-synced frames in a single pass.
• Figure 3: Lips Encoder. A frozen expression encoder with an MLP projector and a mouth-crop CNN produce an 8D+4D lips vector. A distilled ResNet-34 replicates this mapping on inference.
• Figure C.1: Human Preference Evaluation. We conducted a user study comparing FlashLips against randomly selected baseline models. Participants indicated their preference across two criteria: Visual Quality and Lip Sync. The chart displays the number of responses favoring our model (Ours), the competing model (Other), or neither (Same).
• Figure C.2: Limitations. Examples illustrating typical failure cases under challenging conditions, including generating facial hair and teeth details, occlusions, and artifacts caused by the SDXL VAE.