Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Multimodal Online Federated Learning with Modality Missing in Internet of Things

Heqiang Wang, Xiang Liu, Xiaoxiong Zhong, Lixing Chen, Fangming Liu, Weizhe Zhang

TL;DR

This work addresses learning across distributed IoT devices that generate streaming, multimodal data while facing frequent modality missing. It introduces Multimodal Online Federated Learning (MMO-FL) and a Prototypical Modality Mitigation (PMM) algorithm to compensate for missing modalities using online prototypes. Theoretical regret analyses for three scenarios show sublinear regret under suitable learning rates, with missing modalities introducing additional terms dependent on the missing-rate parameter β. Empirical validation on UCI-HAR and MVSA-Single demonstrates that PMM consistently outperforms baselines and can even surpass the ideal full-modality setting as prototypes refine over time, highlighting PMM’s practical impact for robust, privacy-preserving edge learning in IoT.

Abstract

The Internet of Things (IoT) ecosystem generates vast amounts of multimodal data from heterogeneous sources such as sensors, cameras, and microphones. As edge intelligence continues to evolve, IoT devices have progressed from simple data collection units to nodes capable of executing complex computational tasks. This evolution necessitates the adoption of distributed learning strategies to effectively handle multimodal data in an IoT environment. Furthermore, the real-time nature of data collection and limited local storage on edge devices in IoT call for an online learning paradigm. To address these challenges, we introduce the concept of Multimodal Online Federated Learning (MMO-FL), a novel framework designed for dynamic and decentralized multimodal learning in IoT environments. Building on this framework, we further account for the inherent instability of edge devices, which frequently results in missing modalities during the learning process. We conduct a comprehensive theoretical analysis under both complete and missing modality scenarios, providing insights into the performance degradation caused by missing modalities. To mitigate the impact of modality missing, we propose the Prototypical Modality Mitigation (PMM) algorithm, which leverages prototype learning to effectively compensate for missing modalities. Experimental results on two multimodal datasets further demonstrate the superior performance of PMM compared to benchmarks.

Multimodal Online Federated Learning with Modality Missing in Internet of Things

TL;DR

This work addresses learning across distributed IoT devices that generate streaming, multimodal data while facing frequent modality missing. It introduces Multimodal Online Federated Learning (MMO-FL) and a Prototypical Modality Mitigation (PMM) algorithm to compensate for missing modalities using online prototypes. Theoretical regret analyses for three scenarios show sublinear regret under suitable learning rates, with missing modalities introducing additional terms dependent on the missing-rate parameter β. Empirical validation on UCI-HAR and MVSA-Single demonstrates that PMM consistently outperforms baselines and can even surpass the ideal full-modality setting as prototypes refine over time, highlighting PMM’s practical impact for robust, privacy-preserving edge learning in IoT.

Abstract

The Internet of Things (IoT) ecosystem generates vast amounts of multimodal data from heterogeneous sources such as sensors, cameras, and microphones. As edge intelligence continues to evolve, IoT devices have progressed from simple data collection units to nodes capable of executing complex computational tasks. This evolution necessitates the adoption of distributed learning strategies to effectively handle multimodal data in an IoT environment. Furthermore, the real-time nature of data collection and limited local storage on edge devices in IoT call for an online learning paradigm. To address these challenges, we introduce the concept of Multimodal Online Federated Learning (MMO-FL), a novel framework designed for dynamic and decentralized multimodal learning in IoT environments. Building on this framework, we further account for the inherent instability of edge devices, which frequently results in missing modalities during the learning process. We conduct a comprehensive theoretical analysis under both complete and missing modality scenarios, providing insights into the performance degradation caused by missing modalities. To mitigate the impact of modality missing, we propose the Prototypical Modality Mitigation (PMM) algorithm, which leverages prototype learning to effectively compensate for missing modalities. Experimental results on two multimodal datasets further demonstrate the superior performance of PMM compared to benchmarks.

Paper Structure

This paper contains 25 sections, 3 theorems, 20 equations, 8 figures, 1 table, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Online Federated Learning
  4. Multimodal Federated Learning
  5. Prototype Federated Learning
  6. System Model
  7. The Workflow of MMO-FL with Missing Modalities
  8. Client - New Data Collect
  9. Client - Local Model Update
  10. Client - Local Model Upload
  11. Server - Global Model Update
  12. Theoretical Analysis
  13. MMO-FL with local iteration $E=1$ and without modality missing
  14. MMO-FL with local iterations $E>1$ and without modality missing
  15. MMO-FL with local iterations $E>1$ and modality missing
  16. ...and 10 more sections

Key Result

Theorem 1

Under Assumption 1-2, MMO-FL with local iterations $E=1$ and excluding the impact of modality missing, achieves the following regret bound:

Figures (8)

  • Figure 1: IoT-Based MMO-FL with Modality Missing
  • Figure 2: The time diagram of MMO-FL in one global round
  • Figure 3: Illustration of PMM: Online Prototypes Construction (OPC) involves generating prototypes from continuously evolving data throughout the MMO-FL learning process, ensuring a dynamic representation of each modality. Online Prototypes Substitution (OPS), on the other hand, leverages these prototypes to compensate for missing modalities in real-time, enabling effective modality reconstruction for clients encountering incomplete modality data.
  • Figure 4: Performance comparison of proposed algorithm and benchmarks with modality missing.
  • Figure 5: Performance comparison of proposed algorithm with different modality missing rate.
  • ...and 3 more figures

Theorems & Definitions (6)

  • Theorem 1
  • proof
  • Theorem 2
  • proof
  • Theorem 3
  • proof