Learning Online Scale Transformation for Talking Head Video Generation

TL;DR

This work tackles facial scale mismatch in one-shot talking head video generation by introducing OSTNet, an online scale-aware reenactment network. It combines a scale transformation module that uses detected keypoints to rectify the driving face to the source scale with a scale-embedding generator that injects the learned scale code $z$ into every generation layer, enabling anchor-frame-free, scale-consistent motion transfer. The method is trained end-to-end with expression-preserved augmentation and a multi-term loss, and shows superior performance on VoxCeleb1 and HDTF for both same-scale and cross-identity reenactment, as well as robustness to varying facial scales. The results indicate that online scale rectification and multi-layer scale propagation significantly improve realism and identity preservation, with practical impact for scalable, anchor-free talking head generation in real-world applications.

Abstract

One-shot talking head video generation uses a source image and driving video to create a synthetic video where the source person's facial movements imitate those of the driving video. However, differences in scale between the source and driving images remain a challenge for face reenactment. Existing methods attempt to locate a frame in the driving video that aligns best with the source image, but imprecise alignment can result in suboptimal outcomes. To this end, we introduce a scale transformation module that can automatically adjust the scale of the driving image to fit that of the source image, by using the information of scale difference maintained in the detected keypoints of the source image and the driving frame. Furthermore, to keep perceiving the scale information of faces during the generation process, we incorporate the scale information learned from the scale transformation module into each layer of the generation process to produce a final result with an accurate scale. Our method can perform accurate motion transfer between the two images without any anchor frame, achieved through the contributions of the proposed online scale transformation facial reenactment network. Extensive experiments have demonstrated that our proposed method adjusts the scale of the driving face automatically according to the source face, and generates high-quality faces with an accurate scale in the cross-identity facial reenactment.

Figures (12)

• Figure 1: Methods in hong2022depthsiarohin2019first are sensitive to driving face scale (the first row) and rely on offline searching for an anchor frame. However, imperfect alignment can result in suboptimal outputs (the second row). Our method adjusts the driving facial scale to match the source face, eliminating the need for anchor frame searching and producing high-quality animated results (the last row).
• Figure 2: The framework of our OSTNet. We proposed a scale transformation module to align the scale of the source face and the driving face to produce a scale rectified image $I_{\hat{D}}$ using the scale information extracted from both keypoints of the source image $I_S$ and the driving image $I_{D}$. We embed the scale information $z$ learned from the scale transformation module into the generation process to produce the result with a more accurate scale. In this way, our method can automatically adjust the scale of driving face online and produce results with an accurate scale. We utilize the reconstruction loss $\mathcal{L}_{rec}$ to facilitate learning our scale transformation module and the whole network.
• Figure 3: Illustration of the proposed scale transformation module. The symbol ⓦ denotes the warping operation, and the ⓒ represents the concatenation. Inspired by shi2016robust, we utilize the scale information maintained in the detected keypoints to predict the fiducial points. Then we feed the fiducial points into the grid generator to produce the deformation map.
• Figure 4: Illustration of the scale embedding in the generation process. The symbol ⓦ denotes the warping operation, and the symbol + represents the element-wise addition operator.
• Figure 5: Qualitative comparisons of same-scale same-identity reenactment on the VoxCeleb1 dataset