Dubbing for Everyone: Data-Efficient Visual Dubbing using Neural Rendering Priors

TL;DR

Visual dubbing demands high visual quality, generalisability across actors, scalability, and recognisability of speaking style. The authors introduce Dubbing for Everyone, a hybrid approach that learns a data-efficient, shared deferred neural rendering prior across identities and augments it with actor-specific neural textures, enabling high-quality dubbing from just a few seconds of data. The method achieves state-of-the-art visual quality and recognisability, demonstrated through user studies, and trains faster than per-identity models due to prior-based adaptation and texture specialization. This work reduces data requirements and broadens accessibility for dubbing all actors, from A-listers to background talent, while providing a framework for ethical considerations and detector-assisted safeguards. The approach has practical impact for multilingual media production and localization, offering a scalable path to realistic, actor-aware lip-sync across diverse identities.

Abstract

Visual dubbing is the process of generating lip motions of an actor in a video to synchronise with given audio. Recent advances have made progress towards this goal but have not been able to produce an approach suitable for mass adoption. Existing methods are split into either person-generic or person-specific models. Person-specific models produce results almost indistinguishable from reality but rely on long training times using large single-person datasets. Person-generic works have allowed for the visual dubbing of any video to any audio without further training, but these fail to capture the person-specific nuances and often suffer from visual artefacts. Our method, based on data-efficient neural rendering priors, overcomes the limitations of existing approaches. Our pipeline consists of learning a deferred neural rendering prior network and actor-specific adaptation using neural textures. This method allows for $\textbf{high-quality visual dubbing with just a few seconds of data}$, that enables video dubbing for any actor - from A-list celebrities to background actors. We show that we achieve state-of-the-art in terms of $\textbf{visual quality}$ and $\textbf{recognisability}$ both quantitatively, and qualitatively through two user studies. Our prior learning and adaptation method $\textbf{generalises to limited data}$ better and is more $\textbf{scalable}$ than existing person-specific models. Our experiments on real-world, limited data scenarios find that our model is preferred over all others. The project page may be found at https://dubbingforeveryone.github.io/

Figures (8)

• Figure 1: Our method, Dubbing for Everyone allows for the reconstruction of lip movements when dubbing video from one language to another. Our approach overcomes the limitations of person-generic and person-specific visual dubbing models, to produce a model that is of high visual quality, recognisable, generalisable and scalable
• Figure 2: The pipeline of our method. We first apply preprocessing to our dataset (\ref{['sec:preprocess']}) to obtain 3D reconstructions, tightly and stably cropped to the face. We next obtain person-generic audio-to-expression and neural rendering models using multiple subjects (\ref{['sec:generic']}). Given a new subject, we then finetune both models for the given subject (\ref{['sec:finetune']}).
• Figure 3: The architecture of our model. We take the UV rasterisations of each frame in a window of length T and sample a neural texture. Using a mask, we combine this with the real frame (\ref{['fig:NetworkInput']}). These are concatenated with random reference frames of the same person and stacked across the channel dimension. A StyleGAN2-based guan2023stylesyncKarras2019stylegan2 generator is then used to convert these into T photorealistic frames.
• Figure 4: The input to the image-to-image network. We sample the neural texture onto the rasterised mesh and use the mesh to estimate a mask. The input is computed using this mask, the target frame, and the rasterised texture.
• Figure 5: Qualitative comparisons to state-of-the-art person-specific models. We compare to GeneFace Ye2023GeneFaceGA, RAD-NeRF tang2022radnerf and the baseline model trained from scratch.