Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

ModalityMirror: Improving Audio Classification in Modality Heterogeneity Federated Learning with Multimodal Distillation

Tiantian Feng, Tuo Zhang, Salman Avestimehr, Shrikanth S. Narayanan

TL;DR

ModalityMirror tackles modality heterogeneity in audiovisual Federated Learning by first learning a multimodal model via modality-aware FL, then distilling multimodal knowledge into an unimodal audio model through federated distillation. The two-phase framework enables audio-only clients to benefit from visual information without requiring them to access multi-modal data, addressing privacy and data-heterogeneity constraints. Empirical results on UCF101 and ActivityNet show consistent audio performance gains over state-of-the-art FL baselines, particularly as visual modality availability fluctuates, and ablations reveal improved label discrimination where audio cues alone are weak. The approach offers a practical pathway to exploit the full modality spectrum in privacy-preserving distributed learning, with some limitations on acoustically-dominant labels that warrant further investigation.

Abstract

Multimodal Federated Learning frequently encounters challenges of client modality heterogeneity, leading to undesired performances for secondary modality in multimodal learning. It is particularly prevalent in audiovisual learning, with audio is often assumed to be the weaker modality in recognition tasks. To address this challenge, we introduce ModalityMirror to improve audio model performance by leveraging knowledge distillation from an audiovisual federated learning model. ModalityMirror involves two phases: a modality-wise FL stage to aggregate uni-modal encoders; and a federated knowledge distillation stage on multi-modality clients to train an unimodal student model. Our results demonstrate that ModalityMirror significantly improves the audio classification compared to the state-of-the-art FL methods such as Harmony, particularly in audiovisual FL facing video missing. Our approach unlocks the potential for exploiting the diverse modality spectrum inherent in multi-modal FL.

ModalityMirror: Improving Audio Classification in Modality Heterogeneity Federated Learning with Multimodal Distillation

TL;DR

ModalityMirror tackles modality heterogeneity in audiovisual Federated Learning by first learning a multimodal model via modality-aware FL, then distilling multimodal knowledge into an unimodal audio model through federated distillation. The two-phase framework enables audio-only clients to benefit from visual information without requiring them to access multi-modal data, addressing privacy and data-heterogeneity constraints. Empirical results on UCF101 and ActivityNet show consistent audio performance gains over state-of-the-art FL baselines, particularly as visual modality availability fluctuates, and ablations reveal improved label discrimination where audio cues alone are weak. The approach offers a practical pathway to exploit the full modality spectrum in privacy-preserving distributed learning, with some limitations on acoustically-dominant labels that warrant further investigation.

Abstract

Multimodal Federated Learning frequently encounters challenges of client modality heterogeneity, leading to undesired performances for secondary modality in multimodal learning. It is particularly prevalent in audiovisual learning, with audio is often assumed to be the weaker modality in recognition tasks. To address this challenge, we introduce ModalityMirror to improve audio model performance by leveraging knowledge distillation from an audiovisual federated learning model. ModalityMirror involves two phases: a modality-wise FL stage to aggregate uni-modal encoders; and a federated knowledge distillation stage on multi-modality clients to train an unimodal student model. Our results demonstrate that ModalityMirror significantly improves the audio classification compared to the state-of-the-art FL methods such as Harmony, particularly in audiovisual FL facing video missing. Our approach unlocks the potential for exploiting the diverse modality spectrum inherent in multi-modal FL.
Paper Structure (20 sections, 2 equations, 3 figures, 1 table, 2 algorithms)

This paper contains 20 sections, 2 equations, 3 figures, 1 table, 2 algorithms.

Table of Contents

  1. Introduction
  2. Related Study
  3. Method
  4. Problem Formulation
  5. ModalityMirror For Audio-visual Recognition
  6. Overview
  7. Modality-Aware Federated Learning.
  8. Federated Distillation
  9. Datasets and Experimental Details
  10. Audio-visual Datasets
  11. Model Details
  12. Evaluation Baselines
  13. Evaluation Metrics
  14. Experiments
  15. End-to-End Performance
  16. ...and 5 more sections

Figures (3)

  • Figure 1: System Architecture of ModalityMirror. ModalityMirror comprises two distinct stages: modality-aware federated learning across all nodes, followed by federated distillation leveraging data-rich nodes.
  • Figure 2: Comparative Analysis of Top 10 relative performance changes across individual labels between Harmony and ModalityMirror with UCF101 and ActivityNet dataset in audio modality respectively, where each bar represents the difference in F1 scores for a specific label.
  • Figure 3: Accuracy performance of ModalityMirror on UCF101 under various video modality missing rates.