Contrastive Learning-based Chaining-Cluster for Multilingual Voice-Face Association

TL;DR

This work tackles face-voice association in multilingual environments by coupling a supervised cross-contrastive (SCC) learning framework with a chaining-cluster post-processing step. The SCC stage aligns voice and face embeddings in a shared space using a two-branch Transformer-based architecture with a shared layer and a cross-contrastive loss, while initial test scores are derived from $1-\cos(E_v(v_i),E_f(f_i))$. The chaining-cluster phase then clusters samples by gender and identity, builds high-confidence prototypes, computes cross-modal prototype similarities, and refines scores via gender-mismatch penalties and prototype rewards, thereby reducing the impact of outliers. Evaluations on MAV-Celeb for FAME 2024 demonstrate strong performance, achieving 2nd place and showing robustness to both heard and unheard language scenarios; the ablations confirm the utility of the post-processing step. The work provides a practical, outlier-resilient solution for multilingual voice-face association and releases code to enable reproducibility and extension.

Abstract

The innate correlation between a person's face and voice has recently emerged as a compelling area of study, especially within the context of multilingual environments. This paper introduces our novel solution to the Face-Voice Association in Multilingual Environments (FAME) 2024 challenge, focusing on a contrastive learning-based chaining-cluster method to enhance face-voice association. This task involves the challenges of building biometric relations between auditory and visual modality cues and modelling the prosody interdependence between different languages while addressing both intrinsic and extrinsic variability present in the data. To handle these non-trivial challenges, our method employs supervised cross-contrastive (SCC) learning to establish robust associations between voices and faces in multi-language scenarios. Following this, we have specifically designed a chaining-cluster-based post-processing step to mitigate the impact of outliers often found in unconstrained in the wild data. We conducted extensive experiments to investigate the impact of language on face-voice association. The overall results were evaluated on the FAME public evaluation platform, where we achieved 2nd place. The results demonstrate the superior performance of our method, and we validate the robustness and effectiveness of our proposed approach. Code is available at https://github.com/colaudiolab/FAME24_solution.

Figures (3)

• Figure 1: Our method consists of a supervised cross-contrastive learning phase and a chaining-cluster phase. The first phase accepts a pair of voice-face and outputs their similarity score (We use V1 unheard as an illustrative example). The chaining-cluster phase is employed only during the testing phase, as a post-processing step to refine the initial score from the first phase.
• Figure 2: The blue line represents the total $L_2$ distance of each sample to its cluster center, and the red line represents the distance difference between adjacent cluster numbers.
• Figure 3: Cross-modal prototype similarity matrix. The raw represents voice prototypes, and the column represents the face prototypes.