XM-ALIGN: Unified Cross-Modal Embedding Alignment for Face-Voice Association
Zhihua Fang, Shumei Tao, Junxu Wang, Liang He
TL;DR
This work tackles multilingual cross-modal face–voice verification by introducing XM-ALIGN, a unified framework that couples explicit embedding alignment (MSE between face and voice embeddings) with implicit alignment via a shared classifier. Two parallel encoders (ResNet-18 for faces and ECAPA-TDNN for voices) produce embeddings that are jointly optimized through a combined loss, enabling robust cross-language verification. Experimental results on MAV-Celeb show that explicit alignment plus a shared classifier improves performance, with score fusion across top models further boosting results, especially in unheard-language scenarios. The approach demonstrates strong cross-modal verification gains and provides a reproducible release path via code.
Abstract
This paper introduces our solution, XM-ALIGN (Unified Cross-Modal Embedding Alignment Framework), proposed for the FAME challenge at ICASSP 2026. Our framework combines explicit and implicit alignment mechanisms, significantly improving cross-modal verification performance in both "heard" and "unheard" languages. By extracting feature embeddings from both face and voice encoders and jointly optimizing them using a shared classifier, we employ mean squared error (MSE) as the embedding alignment loss to ensure tight alignment between modalities. Additionally, data augmentation strategies are applied during model training to enhance generalization. Experimental results show that our approach demonstrates superior performance on the MAV-Celeb dataset. The code will be released at https://github.com/PunkMale/XM-ALIGN.