Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Learning Reconfigurable Representations for Multimodal Federated Learning with Missing Data

Duong M. Nguyen, Trong Nghia Hoang, Thanh Trung Huynh, Quoc Viet Hung Nguyen, Phi Le Nguyen

TL;DR

PEPSY tackles multimodal federated learning when clients experience both cross-client modality heterogeneity and intra-modal feature missingness. It introduces a data-missing profile of embedding controls at the client side to reconfigure the shared representation toward each local context, with server-side PFPT-based clustering to align profiles across clients. A theoretical bound links missing-pattern robustness to an alignment loss L_ds, and empirical results on PTBXL and Sleep-EDF demonstrate up to substantial gains under severe incompleteness, validating both the approach and its robustness to varied missing patterns. This framework advances privacy-preserving MMFL by enabling flexible, context-aware representation reconfiguration without data sharing.

Abstract

Multimodal federated learning in real-world settings often encounters incomplete and heterogeneous data across clients. This results in misaligned local feature representations that limit the effectiveness of model aggregation. Unlike prior work that assumes either differing modality sets without missing input features or a shared modality set with missing features across clients, we consider a more general and realistic setting where each client observes a different subset of modalities and might also have missing input features within each modality. To address the resulting misalignment in learned representations, we propose a new federated learning framework featuring locally adaptive representations based on learnable client-side embedding controls that encode each client's data-missing patterns. These embeddings serve as reconfiguration signals that align the globally aggregated representation with each client's local context, enabling more effective use of shared information. Furthermore, the embedding controls can be algorithmically aggregated across clients with similar data-missing patterns to enhance the robustness of reconfiguration signals in adapting the global representation. Empirical results on multiple federated multimodal benchmarks with diverse data-missing patterns across clients demonstrate the efficacy of the proposed method, achieving up to 36.45\% performance improvement under severe data incompleteness. The method is also supported by a theoretical analysis with an explicit performance bound that matches our empirical observations. Our source codes are provided at https://github.com/nmduonggg/PEPSY

Learning Reconfigurable Representations for Multimodal Federated Learning with Missing Data

TL;DR

PEPSY tackles multimodal federated learning when clients experience both cross-client modality heterogeneity and intra-modal feature missingness. It introduces a data-missing profile of embedding controls at the client side to reconfigure the shared representation toward each local context, with server-side PFPT-based clustering to align profiles across clients. A theoretical bound links missing-pattern robustness to an alignment loss L_ds, and empirical results on PTBXL and Sleep-EDF demonstrate up to substantial gains under severe incompleteness, validating both the approach and its robustness to varied missing patterns. This framework advances privacy-preserving MMFL by enabling flexible, context-aware representation reconfiguration without data sharing.

Abstract

Multimodal federated learning in real-world settings often encounters incomplete and heterogeneous data across clients. This results in misaligned local feature representations that limit the effectiveness of model aggregation. Unlike prior work that assumes either differing modality sets without missing input features or a shared modality set with missing features across clients, we consider a more general and realistic setting where each client observes a different subset of modalities and might also have missing input features within each modality. To address the resulting misalignment in learned representations, we propose a new federated learning framework featuring locally adaptive representations based on learnable client-side embedding controls that encode each client's data-missing patterns. These embeddings serve as reconfiguration signals that align the globally aggregated representation with each client's local context, enabling more effective use of shared information. Furthermore, the embedding controls can be algorithmically aggregated across clients with similar data-missing patterns to enhance the robustness of reconfiguration signals in adapting the global representation. Empirical results on multiple federated multimodal benchmarks with diverse data-missing patterns across clients demonstrate the efficacy of the proposed method, achieving up to 36.45\% performance improvement under severe data incompleteness. The method is also supported by a theoretical analysis with an explicit performance bound that matches our empirical observations. Our source codes are provided at https://github.com/nmduonggg/PEPSY
Paper Structure (34 sections, 1 theorem, 37 equations, 9 figures, 11 tables)

This paper contains 34 sections, 1 theorem, 37 equations, 9 figures, 11 tables.

Table of Contents

  1. Introduction
  2. Multimodal Federated Learning (MMFL) with Missing Data
  3. Problem Formulation and Method Overview
  4. Client Design
  5. Data-Missing Representations
  6. Embedding Controls Selection
  7. Reconfiguration Regularization and Modality Fusion
  8. Server Aggregation
  9. Theoretical Analysis
  10. Empirical Evaluation
  11. Experimental Settings
  12. Performance under Similar Missing Statistics between Training and Testing
  13. Performance under Different Missing Statistics between Training and Testing
  14. Ablation Studies
  15. Related Works
  16. ...and 19 more sections

Key Result

Theorem 3.1

Let $x \in \mathcal{D}$ be an arbitrary instance with a missing modality pattern $\mathcal{S} \subset \mathcal{M}$, where $\mathcal{M}$ denotes the full set of modalities. Suppose $y_x^{\mathcal{S}}$ and $y_x^{\emptyset}$ represent the model’s outputs at test time when $x$ is missing modalities in $

Figures (9)

  • Figure 1: From left to right: (a) Performance comparison showing that FedAvg degrades rapidly with increasing missing data, while our framework PEPSY remains robust; (b) Illustration of two types of data-missing events in MMFL systems: (1) missing modalities and (2) missing input features; (c) Conceptual illustration highlighting the key distinction between our approach and prior work (see Section \ref{['sec: method']}).
  • Figure 2: Overview of the overall server-client workflow of PEPSY and its client design.
  • Figure 2: Performance of baselines under various missing statistics, where the missing statistics of the clients and server are different.
  • Figure 3: Impact of alignment loss on performance deviation.
  • Figure 4: Modality representations of different methods under multiple missing scenarios. We train and provide t-SNE 2D visualizations of modality representations constructed by three methods, including our proposal, in different $p_m/p_s$ settings. All experiments are conducted on EDF dataset, nonIID setting.
  • ...and 4 more figures

Theorems & Definitions (2)

  • Theorem 3.1
  • Remark D.3