MoDE: Mixture of Diffusion Experts for Any Occluded Face Recognition

TL;DR

MoDE introduces a diffusion-based occluded face recognition framework that enlists multiple repainted face reconstructions as a dynamic ensemble. An Identity-Gating Network assigns adaptive weights to each reconstruction, enabling a weighted fusion of identity information via $S_{X} = \sum_{i=0}^{n} w(x_i) s(x_i)$ and $W(X) = \text{softmax}(W_{g} \cdot F(X) + b_{g})$. The model is plug-and-play with existing face recognizers and demonstrates improved accuracy on artificial and real occlusions across MS1M, LFW, CelebA, OVF, and WWCF, highlighting the value of diverse, identity-preserving repaints for occlusion robustness. By jointly optimizing diffusion loss and ID-Gate cross-entropy, MoDE effectively mitigates occlusion-induced information loss and enhances recognition reliability in real-world scenarios.

Abstract

With the continuous impact of epidemics, people have become accustomed to wearing masks. However, most current occluded face recognition (OFR) algorithms lack prior knowledge of occlusions, resulting in poor performance when dealing with occluded faces of varying types and severity in reality. Recognizing occluded faces is still a significant challenge, which greatly affects the convenience of people's daily lives. In this paper, we propose an identity-gated mixture of diffusion experts (MoDE) for OFR. Each diffusion-based generative expert estimates one possible complete image for occluded faces. Considering the random sampling process of the diffusion model, which introduces inevitable differences and variations between the inpainted faces and the real ones. To ensemble effective information from multi-reconstructed faces, we introduce an identity-gating network to evaluate the contribution of each reconstructed face to the identity and adaptively integrate the predictions in the decision space. Moreover, our MoDE is a plug-and-play module for most existing face recognition models. Extensive experiments on three public face datasets and two datasets in the wild validate our advanced performance for various occlusions in comparison with the competing methods.

Figures (7)

• Figure 1: To demonstrate the adaptability of our method for multi-occluded face recognition, we visualize some face images from the LFW dataset with six different face occlusions, each image is repainted below. The right part shows the accuracy comparison between the baseline and our method.
• Figure 2: The framework of our MoDE. Firstly, MoDE reconstructs the occluded image and produces n repainted images. Then, it extracts the features from the repainted images and the original occluded image, which are then inputted into ID-Gate to output the weight vector. Finally, we calculate the similarity matrix of each image and multiply it with the weight vector to generate a weighted similarity matrix, thereby obtaining the recognition result.
• Figure 3: Face Repainting. Repaint modifies the standard denoising process to condition on the given image content. In each step, it samples the known region (top) from the input and the repainted part from the output (bottom).
• Figure 4: Visualization of similarity distributions with t-SNE van2008visualizing and following normalization. Different markers with colors represent different classes.
• Figure 5: Visualization of face repaint error. The first three columns represent normal faces, occluded faces and repainted faces, respectively. The last three columns represent heatmaps indicating the degree of information loss between GT and Occluded, RF, and MoDE, respectively.