Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Contrastive Language-Image Learning with Augmented Textual Prompts for 3D/4D FER Using Vision-Language Model

Muzammil Behzad, Guoying Zhao

TL;DR

AffectVLM introduces a multiview vision–language framework for 3D/4D FER by learning a shared embedding space that fuses frontal, left, and right views with augmented textual prompts. The model integrates a gradient-friendly, learnable-margin loss combining multiview contrastive and triplet components, and employs mixed view augmentation to expand data diversity, all within a CLIP-based backbone supporting ViT/ResNet. It also demonstrates scalable distributed training and a Streamlit-based interactive inference interface, achieving state-of-the-art results across BU-3DFE, Bosphorus, BU-4DFE, and BP4D-Spontaneous datasets, including spontaneous expressions and cross-dataset evaluations. The work highlights the effectiveness of joint visual–lingual representations for robust FER under varied viewpoints and contexts, with practical implications for real-time affective computing and extension to other facial analysis tasks.

Abstract

In this paper, we introduce AffectVLM, a vision-language model designed to integrate multiviews for a semantically rich and visually comprehensive understanding of facial emotions from 3D/4D data. To effectively capture visual features, we propose a joint representation learning framework paired with a novel gradient-friendly loss function that accelerates model convergence towards optimal feature representation. Additionally, we introduce augmented textual prompts to enhance the model's linguistic capabilities and employ mixed view augmentation to expand the visual dataset. We also develop a Streamlit app for a real-time interactive inference and enable the model for distributed learning. Extensive experiments validate the superior performance of AffectVLM across multiple benchmarks.

Contrastive Language-Image Learning with Augmented Textual Prompts for 3D/4D FER Using Vision-Language Model

TL;DR

AffectVLM introduces a multiview vision–language framework for 3D/4D FER by learning a shared embedding space that fuses frontal, left, and right views with augmented textual prompts. The model integrates a gradient-friendly, learnable-margin loss combining multiview contrastive and triplet components, and employs mixed view augmentation to expand data diversity, all within a CLIP-based backbone supporting ViT/ResNet. It also demonstrates scalable distributed training and a Streamlit-based interactive inference interface, achieving state-of-the-art results across BU-3DFE, Bosphorus, BU-4DFE, and BP4D-Spontaneous datasets, including spontaneous expressions and cross-dataset evaluations. The work highlights the effectiveness of joint visual–lingual representations for robust FER under varied viewpoints and contexts, with practical implications for real-time affective computing and extension to other facial analysis tasks.

Abstract

In this paper, we introduce AffectVLM, a vision-language model designed to integrate multiviews for a semantically rich and visually comprehensive understanding of facial emotions from 3D/4D data. To effectively capture visual features, we propose a joint representation learning framework paired with a novel gradient-friendly loss function that accelerates model convergence towards optimal feature representation. Additionally, we introduce augmented textual prompts to enhance the model's linguistic capabilities and employ mixed view augmentation to expand the visual dataset. We also develop a Streamlit app for a real-time interactive inference and enable the model for distributed learning. Extensive experiments validate the superior performance of AffectVLM across multiple benchmarks.
Paper Structure (17 sections, 2 equations, 3 figures, 4 tables)

This paper contains 17 sections, 2 equations, 3 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Motivations
  3. Contributions
  4. AffectVLM: Affective Vision-Language Model
  5. Learning Joint Embedding Space Representations
  6. Augmented Textual Prompts
  7. Mixed View Augmentation
  8. Gradient-friendly Loss Function with Learnable Margin
  9. Performance Scalability with Distributed Leaning
  10. Interactive Inference with Streamlit
  11. Results and Discussions
  12. Performance on 3D FER
  13. Performance on 4D FER
  14. Towards Spontaneous 4D FER
  15. Performance Upgrade with Distributed Learning
  16. ...and 2 more sections

Figures (3)

  • Figure 1: Overview of our proposed AffectVLM: Affective Vision-Language Model.
  • Figure 2: Performance comparisons of distributed learning.
  • Figure 3: Ablation study showing the impact of individual components.