Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

VM-BeautyNet: A Synergistic Ensemble of Vision Transformer and Mamba for Facial Beauty Prediction

Djamel Eddine Boukhari

TL;DR

This work addresses Facial Beauty Prediction (FBP), a subjective and holistic computer vision task, by introducing VM-BeautyNet, a heterogeneous ensemble that fuses a Vision Transformer (ViT) with a Vision Mamba backbone to capture global facial structure and long-range sequential features. The model uses a learnable fusion module to combine predictions from both branches and is trained end-to-end under a Mean Squared Error objective. On the SCUT-FBP5500 benchmark, VM-BeautyNet achieves state-of-the-art results (PC 0.9212, MAE 0.2085, RMSE 0.2698) and is supported by Grad-CAM analyses that reveal complementary backbone behavior, providing interpretability. The work highlights a powerful architectural paradigm for computational aesthetics, with implications for fairer, context-aware, and multimodal extensions in future research.

Abstract

Facial Beauty Prediction (FBP) is a complex and challenging computer vision task, aiming to model the subjective and intricate nature of human aesthetic perception. While deep learning models, particularly Convolutional Neural Networks (CNNs), have made significant strides, they often struggle to capture the global, holistic facial features that are critical to human judgment. Vision Transformers (ViT) address this by effectively modeling long-range spatial relationships, but their quadratic complexity can be a bottleneck. This paper introduces a novel, heterogeneous ensemble architecture, \textbf{VM-BeautyNet}, that synergistically fuses the complementary strengths of a Vision Transformer and a Mamba-based Vision model, a recent advancement in State-Space Models (SSMs). The ViT backbone excels at capturing global facial structure and symmetry, while the Mamba backbone efficiently models long-range dependencies with linear complexity, focusing on sequential features and textures. We evaluate our approach on the benchmark SCUT-FBP5500 dataset. Our proposed VM-BeautyNet achieves state-of-the-art performance, with a \textbf{Pearson Correlation (PC) of 0.9212}, a \textbf{Mean Absolute Error (MAE) of 0.2085}, and a \textbf{Root Mean Square Error (RMSE) of 0.2698}. Furthermore, through Grad-CAM visualizations, we provide interpretability analysis that confirms the complementary feature extraction of the two backbones, offering new insights into the model's decision-making process and presenting a powerful new architectural paradigm for computational aesthetics.

VM-BeautyNet: A Synergistic Ensemble of Vision Transformer and Mamba for Facial Beauty Prediction

TL;DR

This work addresses Facial Beauty Prediction (FBP), a subjective and holistic computer vision task, by introducing VM-BeautyNet, a heterogeneous ensemble that fuses a Vision Transformer (ViT) with a Vision Mamba backbone to capture global facial structure and long-range sequential features. The model uses a learnable fusion module to combine predictions from both branches and is trained end-to-end under a Mean Squared Error objective. On the SCUT-FBP5500 benchmark, VM-BeautyNet achieves state-of-the-art results (PC 0.9212, MAE 0.2085, RMSE 0.2698) and is supported by Grad-CAM analyses that reveal complementary backbone behavior, providing interpretability. The work highlights a powerful architectural paradigm for computational aesthetics, with implications for fairer, context-aware, and multimodal extensions in future research.

Abstract

Facial Beauty Prediction (FBP) is a complex and challenging computer vision task, aiming to model the subjective and intricate nature of human aesthetic perception. While deep learning models, particularly Convolutional Neural Networks (CNNs), have made significant strides, they often struggle to capture the global, holistic facial features that are critical to human judgment. Vision Transformers (ViT) address this by effectively modeling long-range spatial relationships, but their quadratic complexity can be a bottleneck. This paper introduces a novel, heterogeneous ensemble architecture, \textbf{VM-BeautyNet}, that synergistically fuses the complementary strengths of a Vision Transformer and a Mamba-based Vision model, a recent advancement in State-Space Models (SSMs). The ViT backbone excels at capturing global facial structure and symmetry, while the Mamba backbone efficiently models long-range dependencies with linear complexity, focusing on sequential features and textures. We evaluate our approach on the benchmark SCUT-FBP5500 dataset. Our proposed VM-BeautyNet achieves state-of-the-art performance, with a \textbf{Pearson Correlation (PC) of 0.9212}, a \textbf{Mean Absolute Error (MAE) of 0.2085}, and a \textbf{Root Mean Square Error (RMSE) of 0.2698}. Furthermore, through Grad-CAM visualizations, we provide interpretability analysis that confirms the complementary feature extraction of the two backbones, offering new insights into the model's decision-making process and presenting a powerful new architectural paradigm for computational aesthetics.
Paper Structure (30 sections, 4 equations, 1 figure, 2 tables, 1 algorithm)

This paper contains 30 sections, 4 equations, 1 figure, 2 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Convolutional Neural Networks for FBP
  4. Vision Transformers in Computational Aesthetics
  5. State-Space Models and Vision Mamba
  6. Ensemble and Hybrid Models
  7. Proposed Methodology
  8. Architectural Components
  9. Vision Transformer Backbone
  10. Vision Mamba Backbone
  11. Fusion Module
  12. Training Procedure and Loss Function
  13. Experiments
  14. Dataset
  15. Evaluation Metrics
  16. ...and 15 more sections

Figures (1)

  • Figure 1: The overall architecture of our proposed VM-BeautyNet. An input image is processed in parallel by two distinct backbones. Branch A (ViT) captures global spatial relationships, while Branch B (Vision Mamba) models sequential features efficiently. The intermediate predictions, $p_{vit}$ and $p_{mamba}$, are then intelligently combined by the Fusion Module $\mathcal{F}$ to produce the final beauty score $\hat{y}$.