SingingBot: An Avatar-Driven System for Robotic Face Singing Performance

TL;DR

SingingBot tackles the challenge of expressive, lip-synced robotic singing by generating avatar-driven driving signals from audio using a diffusion-based video transformer with rich human priors, and transferring these signals to a physical robot through a semantic piecewise mapping. The pipeline first creates a driving avatar video $\mathbf{V}_{0:T-1}$ with $\mathbf{V}_{0:T-1} = \mathcal{D}(\mathbf{A}_{0:T-1}, \mathbf{I}_{ref}, \mathbf{p}_{ref})$, then extracts $\mathbf{B}_{0:T-1} \in \mathbb{R}^{52}$ blendshape coefficients and maps them to robot motor commands via $\Delta{\mathbf{m}}_j = \Psi_j(\beta_j) = \mathbf{w}_{j,k} \cdot \beta_j + \mathbf{c}_{j,k}$, combining to $\mathbf{m} = \mathbf{m}_{rest} + \sum_j \Psi_j(\beta_j)$. The method introduces Emotion Dynamic Range (EDR), defined as the area of the convex hull of poses in the Valence-Arousal space, to quantify emotional breadth, and demonstrates superior lip-sync accuracy and richer emotional expressiveness compared with baselines. By leveraging large-scale human priors and a semantic mapping strategy, SingingBot achieves realistic, emotionally varied robotic singing, enabling more natural empathetic human-robot interaction. The work also provides a detailed hardware-mapping pipeline (BS2Action) and supplementary materials, with public release planned after publication.

Abstract

Equipping robotic faces with singing capabilities is crucial for empathetic Human-Robot Interaction. However, existing robotic face driving research primarily focuses on conversations or mimicking static expressions, struggling to meet the high demands for continuous emotional expression and coherence in singing. To address this, we propose a novel avatar-driven framework for appealing robotic singing. We first leverage portrait video generation models embedded with extensive human priors to synthesize vivid singing avatars, providing reliable expression and emotion guidance. Subsequently, these facial features are transferred to the robot via semantic-oriented mapping functions that span a wide expression space. Furthermore, to quantitatively evaluate the emotional richness of robotic singing, we propose the Emotion Dynamic Range metric to measure the emotional breadth within the Valence-Arousal space, revealing that a broad emotional spectrum is crucial for appealing performances. Comprehensive experiments prove that our method achieves rich emotional expressions while maintaining lip-audio synchronization, significantly outperforming existing approaches.

Figures (8)

• Figure 1: We present SingingBot, a framework to generate robotic face singing performance from synthesized portrait animations. Our method conveys the rich emotions of singing while maintaining obvious lip-audio synchronization.
• Figure 2: The overall pipeline of SingingBot for robotic singing performance. Given a vocal audio and a reference portrait, our method first synthesizes a vivid avatar singing animation using a pretrained video diffusion model. Benefiting from the embedded extensive expression and emotion priors, the avatar animation serves as a reliable driving source that provides expressive facial features for subsequent robotic performance. Through the semantic-oriented piecewise functions, the avatar's facial features are then mapped to the physical robot motion space, achieving coherent and appealing robotic singing performance.
• Figure 3: Visualization of Emotion Dynamic Range. The polygonal area represents the emotional richness of singing performance. Our method achieves significantly better emotional expressiveness.
• Figure 4: Visual comparison of generated robotic singing performances. Our method displays rich emotions while presenting plausible lip shapes. The red box highlights the micro expressions expressing subtle emotions.
• Figure 5: Visual comparison of different singing strategies. Using avatar videos embedded with diffusion priors as driving sources significantly enhances the animation performance of robot singing.