Unsupervised Multiview Contrastive Language-Image Joint Learning with Pseudo-Labeled Prompts Via Vision-Language Model for 3D/4D Facial Expression Recognition

TL;DR

The paper tackles unsupervised learning for 3D/4D facial expression recognition by aligning multiview visual data with textual semantics through a vision-language model. It extends CLIP-like architectures to multiview 3D/4D FER and introduces pseudo-labels generated from prompts to provide semantic anchors for cross-modal alignment. A joint embedding space is learned via a multiview contrastive objective, augmented with a positive-negative multiview pairing strategy and a gradient-friendly loss, and the model is designed for scalable distributed training. Empirical results on BU-3DFE, Bosphorus, BU-4DFE, and BP4D-Spontaneous show state-of-the-art or competitive performance among unsupervised methods and strong performance against supervised baselines with good cross-dataset generalization. The approach is modular and adaptable to other multiview emotion-related tasks, offering a scalable path toward real-world deployment.

Abstract

In this paper, we introduce MultiviewVLM, a vision-language model designed for unsupervised contrastive multiview representation learning of facial emotions from 3D/4D data. Our architecture integrates pseudo-labels derived from generated textual prompts to guide implicit alignment of emotional semantics. To capture shared information across multi-views, we propose a joint embedding space that aligns multiview representations without requiring explicit supervision. We further enhance the discriminability of our model through a novel multiview contrastive learning strategy that leverages stable positive-negative pair sampling. A gradient-friendly loss function is introduced to promote smoother and more stable convergence, and the model is optimized for distributed training to ensure scalability. Extensive experiments demonstrate that MultiviewVLM outperforms existing state-of-the-art methods and can be easily adapted to various real-world applications with minimal modifications.

Figures (7)

• Figure 1: Overview of the vision-language model for emotion recognition: (a) Inference framework leveraging multi-view inputs, (b) Supervised adaptation using labeled target datasets, and (c) Unsupervised joint embedding learning with multi-view positive-negative pairs. We use [CLS] to denote an emotion class token as: "Happy", "Angry", "Disgust", "Fear", "Sad", or "Surprise".
• Figure 2: The architecture of MultiviewVLM for unsupervised 3D/4D emotion recognition. The pseudo-labels are generated using a language model such as GPT, and multiview embeddings (front, left, and right) are extracted via vision encoders. The joint representation learning then aligns the textual and visual embeddings in a shared embedding space, while positive-negative pair learning ensures discriminability and consistency across views. The proposed loss $\mathcal{L}_{\text{multiviewVLM}}$ facilitates optimization in the embedding space, resulting in robust emotion representations. During inference, the multiview images are fed through the vision encoder to extract and aggregate embeddings, which are then matched against precomputed textual embeddings using similarity, and then assigning the most semantically relevant emotion label.
• Figure 3: Ablation comparison of each proposed component of MultiviewVLM.
• Figure 4: Performance comparisons of distributed learning.
• Figure 5: Speedup gains and comparison across various GPU configurations.