Uncertainty-Aware Label Refinement on Hypergraphs for Personalized Federated Facial Expression Recognition

TL;DR

This paper tackles personalized federated facial expression recognition (PF-FER) under privacy constraints, addressing heterogeneity in local data and uncertain labels. It introduces AMY, which combines a hypergraph-based uncertainty estimation (UE) block and a hypergraph-based label propagation (EC) block within each client, augmented by a personalized uncertainty estimator and class-prototype regularization during global training. The approach leverages high-order relationships via HGNNs to produce reliable per-sample uncertainty weights and refined labels, enabling robust local models without sharing raw data. Experimental results on RAF-DB and FERPlus show consistent improvements over state-of-the-art FL and PF-FER methods, demonstrating the advantage of hypergraph modeling for uncertainty estimation and label refinement in privacy-preserving FER.

Abstract

Most facial expression recognition (FER) models are trained on large-scale expression data with centralized learning. Unfortunately, collecting a large amount of centralized expression data is difficult in practice due to privacy concerns of facial images. In this paper, we investigate FER under the framework of personalized federated learning, which is a valuable and practical decentralized setting for real-world applications. To this end, we develop a novel uncertainty-Aware label refineMent on hYpergraphs (AMY) method. For local training, each local model consists of a backbone, an uncertainty estimation (UE) block, and an expression classification (EC) block. In the UE block, we leverage a hypergraph to model complex high-order relationships between expression samples and incorporate these relationships into uncertainty features. A personalized uncertainty estimator is then introduced to estimate reliable uncertainty weights of samples in the local client. In the EC block, we perform label propagation on the hypergraph, obtaining high-quality refined labels for retraining an expression classifier. Based on the above, we effectively alleviate heterogeneous sample uncertainty across clients and learn a robust personalized FER model in each client. Experimental results on two challenging real-world facial expression databases show that our proposed method consistently outperforms several state-of-the-art methods. This indicates the superiority of hypergraph modeling for uncertainty estimation and label refinement on the personalized federated FER task. The source code will be released at https://github.com/mobei1006/AMY.

Figures (5)

• Figure 1: Illustration of the challenges of heterogeneous expression data and heterogeneous sample uncertainty. The uncertainty arises from low-quality/ambiguous expression samples and noisy labels. The degree of sample uncertainty varies for client $i$ and client $j$. The images are taken from the RAF-DB database li2017reliable.
• Figure 2: Overview of our proposed AMY method. Each local model is comprised of a backbone, an uncertainty estimation (UE) block, and an expression classification (EC) block. A personalized uncertainty estimator (PUE) is private to each client and not uploaded to the server, enabling personalized training.
• Figure 3: Comparison of different competitors of the UE blo ck at the different values of $\alpha$ on the RAF-DB and FERPlus databases.
• Figure 4: Comparison of different competitors of the EC block at the different values of $\alpha$ on the RAF-DB and FERPlus databases.
• Figure 5: Visualization of the uncertainty weights estimated by SCN (the first row) and our method AMY (the second row) on the RAF-DB database, where a larger weight indicates a higher degree of uncertainty for a sample.