Sonic: Shifting Focus to Global Audio Perception in Portrait Animation

TL;DR

Sonic tackles the limited global audio perception in talking-face animation by shifting the paradigm to audio-centric priors that govern facial expressions and lip movements. It introduces context-enhanced audio learning to extract intra-clip temporal cues and a motion-decoupled controller to separately manage head motion and expressions, complemented by time-aware position shift fusion to fuse intra-clip information into global inter-clip perception. The approach, evaluated on standard HDTF and CelebV-HQ datasets, outperforms state-of-the-art methods in video quality, lip synchronization, temporal consistency, and motion diversity, while supporting long-duration generation without extra inference cost. This work advances practical, audio-driven portrait animation with robust long-range coherence and expressive variety.

Abstract

The study of talking face generation mainly explores the intricacies of synchronizing facial movements and crafting visually appealing, temporally-coherent animations. However, due to the limited exploration of global audio perception, current approaches predominantly employ auxiliary visual and spatial knowledge to stabilize the movements, which often results in the deterioration of the naturalness and temporal inconsistencies.Considering the essence of audio-driven animation, the audio signal serves as the ideal and unique priors to adjust facial expressions and lip movements, without resorting to interference of any visual signals. Based on this motivation, we propose a novel paradigm, dubbed as Sonic, to {s}hift f{o}cus on the exploration of global audio per{c}ept{i}o{n}.To effectively leverage global audio knowledge, we disentangle it into intra- and inter-clip audio perception and collaborate with both aspects to enhance overall perception.For the intra-clip audio perception, 1). \textbf{Context-enhanced audio learning}, in which long-range intra-clip temporal audio knowledge is extracted to provide facial expression and lip motion priors implicitly expressed as the tone and speed of speech. 2). \textbf{Motion-decoupled controller}, in which the motion of the head and expression movement are disentangled and independently controlled by intra-audio clips. Most importantly, for inter-clip audio perception, as a bridge to connect the intra-clips to achieve the global perception, \textbf{Time-aware position shift fusion}, in which the global inter-clip audio information is considered and fused for long-audio inference via through consecutively time-aware shifted windows. Extensive experiments demonstrate that the novel audio-driven paradigm outperform existing SOTA methodologies in terms of video quality, temporally consistency, lip synchronization precision, and motion diversity.

Figures (6)

• Figure 1: Sonic excels in producing vivid portrait animation videos given a reference image and an audio clip. Beyond its fundamental lip-syncing capabilities, Sonic demonstrates proficiency in creating a diverse spectrum of facial expressions and adaptable head movements. Notably, when dealing with extented videos, Sonic can yield stable and seamless outcomes in a parallel fashion, all while maintaining an unified paradigm focusing on global audio perception.
• Figure 2: Framework of our approach. Sonic processes each clip of the long audio in parallel, shifting to a new context at each time step to progressively fuse inter-clip latent features across global audio perception. In Sonic, we enhance intra-clip temporal audio context learning and decouple motion to improve dynamics.
• Figure 3: Illustration of the proposed time-aware position-shift fusion. The model processes each clip non-overlapping. In next timestep, the model starts from a new position determined by the offset, thereby integrating long-range context. Specifically, the tail latents are filled cyclically from the head.
• Figure 4: Qualitative comparisons with State-of-the-Art talking head generation methods. Due to image does not reflect important sync, naturalness and stability, the full video comparison will be included in supplementary materials as well as comparison with demos from other non-open source works.
• Figure 5: Qualitative results under different styles of portrait images and various types of audio inputs. The images and audios were collected from recent works and the Internet. The upper section presents results for audio input ranging from $20$ seconds to $2$ minutes in duration, while the lower section shows results for longer audio inputs, up to $10$ minutes. Our Sonic demonstrates versatility across various portrait styles and maintains vividness over extended durations.