Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

FedUAF: Uncertainty-Aware Fusion with Reliability-Guided Aggregation for Multimodal Federated Sentiment Analysis

Xianxun Zhu, Zezhong Sun, Imad Rida, Erik Cambria, Junqi Su, Rui Wang, Hui Chen

Abstract

Multimodal sentiment analysis in federated learning environments faces significant challenges due to missing modalities, heterogeneous data distributions, and unreliable client updates. Existing federated approaches often struggle to maintain robust performance under these practical conditions. In this paper, we propose FedUAF, a unified multimodal federated learning framework that addresses these challenges through uncertainty-aware fusion and reliability-guided aggregation. FedUAF explicitly models modality-level uncertainty during local training and leverages client reliability to guide global aggregation, enabling effective learning under incomplete and noisy multimodal data. Extensive experiments on CMU-MOSI and CMU-MOSEI demonstrate that FedUAF consistently outperforms state-of-the-art federated baselines across various missing-modality patterns and Non-IID settings. Moreover, FedUAF exhibits superior robustness against noisy clients, highlighting its potential for real-world multimodal federated applications.

FedUAF: Uncertainty-Aware Fusion with Reliability-Guided Aggregation for Multimodal Federated Sentiment Analysis

Abstract

Multimodal sentiment analysis in federated learning environments faces significant challenges due to missing modalities, heterogeneous data distributions, and unreliable client updates. Existing federated approaches often struggle to maintain robust performance under these practical conditions. In this paper, we propose FedUAF, a unified multimodal federated learning framework that addresses these challenges through uncertainty-aware fusion and reliability-guided aggregation. FedUAF explicitly models modality-level uncertainty during local training and leverages client reliability to guide global aggregation, enabling effective learning under incomplete and noisy multimodal data. Extensive experiments on CMU-MOSI and CMU-MOSEI demonstrate that FedUAF consistently outperforms state-of-the-art federated baselines across various missing-modality patterns and Non-IID settings. Moreover, FedUAF exhibits superior robustness against noisy clients, highlighting its potential for real-world multimodal federated applications.
Paper Structure (21 sections, 12 equations, 3 figures, 1 table, 1 algorithm)

This paper contains 21 sections, 12 equations, 3 figures, 1 table, 1 algorithm.

Table of Contents

  1. Introduction
  2. Problem Formulation
  3. Federated Multimodal Data
  4. Prediction-Level Affective Uncertainty
  5. Federated Learning Objective
  6. Methodology
  7. Model Architecture Overview
  8. Prediction-Level Uncertainty Estimation
  9. Uncertainty-Guided Affective Fusion
  10. Uncertainty-Aware Federated Aggregation
  11. Training Procedure
  12. Experiments
  13. Datasets
  14. Federated Data Partition
  15. Baselines
  16. ...and 6 more sections

Figures (3)

  • Figure 1: Overview of the proposed FedUAF framework.
  • Figure 2: Ablation study of FedUAF under severe missing-modality ($\rho_m=0.8$) and strong data heterogeneity (Non-IID intensity $=1.0$). Results on CMU-MOSI (top) and CMU-MOSEI (bottom) are reported in terms of MAE (mean$\pm$std over 3 runs).
  • Figure 3: Robustness to noisy clients under severe missing-modality ($\rho_m=0.8$) and strong data heterogeneity (Non-IID intensity $=1.0$). Results on CMU-MOSI (top) and CMU-MOSEI (bottom) are reported in terms of MAE.