Electromyography-Informed Facial Expression Reconstruction for Physiological-Based Synthesis and Analysis

TL;DR

EIFER introduces an electromyography-informed facial expression reconstruction framework that robustly handles sEMG occlusion by decoupling facial geometry and appearance through a CycleGAN-like unpaired translation paired with a 3D Morphable Model. It establishes a bidirectional mapping between 3DMM expression parameters and muscle activity, enabling both physiological-based expression synthesis (EMG2Exp) and electrode-free facial EMG (Exp2EMG). The method uses a dual encoder–dual generator architecture with cycle-consistency and geometry-regularization losses, trained in two phases, and validated on a dedicated Mimics And Muscles dataset of synchronized sEMG and facial mimicry. Findings show EIFER achieves photorealistic reconstructions under occlusion, preserves facial geometry across expressions, and enables robust synthesis and EMG estimation, suggesting a new paradigm for multimodal facial analysis with potential extension to other occlusion forms and modalities.

Abstract

The relationship between muscle activity and resulting facial expressions is crucial for various fields, including psychology, medicine, and entertainment. The synchronous recording of facial mimicry and muscular activity via surface electromyography (sEMG) provides a unique window into these complex dynamics. Unfortunately, existing methods for facial analysis cannot handle electrode occlusion, rendering them ineffective. Even with occlusion-free reference images of the same person, variations in expression intensity and execution are unmatchable. Our electromyography-informed facial expression reconstruction (EIFER) approach is a novel method to restore faces under sEMG occlusion faithfully in an adversarial manner. We decouple facial geometry and visual appearance (e.g., skin texture, lighting, electrodes) by combining a 3D Morphable Model (3DMM) with neural unpaired image-to-image translation via reference recordings. Then, EIFER learns a bidirectional mapping between 3DMM expression parameters and muscle activity, establishing correspondence between the two domains. We validate the effectiveness of our approach through experiments on a dataset of synchronized sEMG recordings and facial mimicry, demonstrating faithful geometry and appearance reconstruction. Further, we synthesize expressions based on muscle activity and how observed expressions can predict dynamic muscle activity. Consequently, EIFER introduces a new paradigm for facial electromyography, which could be extended to other forms of multi-modal face recordings.

Figures (36)

• Figure 1: EIFER employs a double encoder-generator architecture in a CycleGAN-like framework cyclegan to reconstruct facial geometry and generate photorealistic appearances with artificially applied and removed sEMG electrodes during Phase One. In Phase Two, EIFER learns the bidirectional mapping between expressions and muscle activity, facilitating physiological-based synthesis and electrode-free facial electromyography. Full arrows denote information flow, while dashed arrows denote information flow by regularization terms.
• Figure 2: Three individuals mimic six basic emotions ekmanArgumentBasicEmotions1992, with synchronized sEMG heat-maps illustrating muscular activity buchner2023using. These images showcase varying expression intensities and executions, emphasizing the need for a robust sEMG occlusion method.
• Figure 3: Facial Geometry: We visualize the estimated face geometry under sEMG occlusion for three individuals mimicking expressions. MC-CycleGAN has no face model and is thus omitted.
• Figure 4: Appearance reconstruction and electrode removal: Among the state-of-the-art methods, only SMIRK fails the reconstruction. MC-CycleGAN and EIFER keep the nuanced features.
• Figure 5: Comparison of Synthesized Facial Expressions: Using all expression encoder models, we synthesize various expressions (on a shape-free face) estimated from recorded muscle activity. For a fair comparison, we evaluate all on the electrode-occluded and the by MC-CycleGAN buchner2023letbuchner2023improved restored recordings. EIFER achieves comparable performance on occluded recordings, whereas other methods struggle to produce accurate results even on occlusion-free faces. Video reconstructions are provided in the supplementary material.