Fuse after Align: Improving Face-Voice Association Learning via Multimodal Encoder

TL;DR

This paper addresses unsupervised learning of voice-face associations and introduces Fuse after Align (FAA), a multimodal-encoder framework that learns cross-modal relations after a contrastive learning stage and a face-voice matching objective. It leverages progressive clustering and effective pair selection to diversify positive pairs and hard negatives, enabling more robust modality alignment. FAA achieves state-of-the-art results on voice-face verification, matching, and retrieval on VoxCeleb-derived data, with tangible gains over prior approaches. By integrating clustering-driven supervision with cross-modal fusion, the work advances practical cross-modal representation learning for multimodal identity understanding.

Abstract

Today, there have been many achievements in learning the association between voice and face. However, most previous work models rely on cosine similarity or L2 distance to evaluate the likeness of voices and faces following contrastive learning, subsequently applied to retrieval and matching tasks. This method only considers the embeddings as high-dimensional vectors, utilizing a minimal scope of available information. This paper introduces a novel framework within an unsupervised setting for learning voice-face associations. By employing a multimodal encoder after contrastive learning and addressing the problem through binary classification, we can learn the implicit information within the embeddings in a more effective and varied manner. Furthermore, by introducing an effective pair selection method, we enhance the learning outcomes of both contrastive learning and the matching task. Empirical evidence demonstrates that our framework achieves state-of-the-art results in voice-face matching, verification, and retrieval tasks, improving verification by approximately 3%, matching by about 2.5%, and retrieval by around 1.3%.

Figures (3)

• Figure 1: FAA achieves state-of-the-art performance on various tasks when comparing to previous works
• Figure 2: An overview of the proposed method: Face and voice data without identity information are input into a face and voice encoder, respectively. The resulting embeddings are then subjected to dual-modality pooling and clustering to generate pseudo-labels. We introduce a face-voice contrastive learning for modality alignment and propose a face-voice matching to train the Multimodal encoder. The aim is to use it to learn the relationships between modalities. The entire training process iteratively repeats the clustering and metric learning steps.
• Figure 3: Examples of training pairs: poor positive pairs may be sampled from the same video segment, where they are very similar in aspects such as environment, attire, and facial expressions. In contrast, diverse positive pairs are sampled from the same individual but feature a variety of background elements. In the case of poor negative pairs, the difference between two distinct individuals can be very pronounced, making it easy for the model to discern that they are not the same person. However, hard negative pairs may involve two individuals who share certain similarities, such as gender and facial hair.