Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

FedMMKT:Co-Enhancing a Server Text-to-Image Model and Client Task Models in Multi-Modal Federated Learning

Ningxin He, Yang Liu, Wei Sun, Xiaozhou Ye, Ye Ouyang, Tiegang Gao, Zehui Zhang

TL;DR

FedMMKT introduces a privacy-preserving multimodal federated learning framework that co-enhances a server text-to-image (T2I) model and client task models using synthetic cross-modal data generated on the server. It employs LabVote label refinement and MultiRepFusion multimodal representation fusion to robustly align knowledge from heterogeneous clients, enabling bidirectional knowledge transfer without sharing raw data or model parameters. Theoretical convergence guarantees under non-IID data and multi-modal alignment errors are provided, and experiments on Flowers102 and Food101 show substantial gains for both clients and the server along with favorable communication costs. Overall, FedMMKT demonstrates a practical, scalable approach to cross-silo knowledge transfer in privacy-sensitive, multimodal FL settings.

Abstract

Text-to-Image (T2I) models have demonstrated their versatility in a wide range of applications. However, adaptation of T2I models to specialized tasks is often limited by the availability of task-specific data due to privacy concerns. On the other hand, harnessing the power of rich multimodal data from modern mobile systems and IoT infrastructures presents a great opportunity. This paper introduces Federated Multi-modal Knowledge Transfer (FedMMKT), a novel framework that enables co-enhancement of a server T2I model and client task-specific models using decentralized multimodal data without compromising data privacy.

FedMMKT:Co-Enhancing a Server Text-to-Image Model and Client Task Models in Multi-Modal Federated Learning

TL;DR

FedMMKT introduces a privacy-preserving multimodal federated learning framework that co-enhances a server text-to-image (T2I) model and client task models using synthetic cross-modal data generated on the server. It employs LabVote label refinement and MultiRepFusion multimodal representation fusion to robustly align knowledge from heterogeneous clients, enabling bidirectional knowledge transfer without sharing raw data or model parameters. Theoretical convergence guarantees under non-IID data and multi-modal alignment errors are provided, and experiments on Flowers102 and Food101 show substantial gains for both clients and the server along with favorable communication costs. Overall, FedMMKT demonstrates a practical, scalable approach to cross-silo knowledge transfer in privacy-sensitive, multimodal FL settings.

Abstract

Text-to-Image (T2I) models have demonstrated their versatility in a wide range of applications. However, adaptation of T2I models to specialized tasks is often limited by the availability of task-specific data due to privacy concerns. On the other hand, harnessing the power of rich multimodal data from modern mobile systems and IoT infrastructures presents a great opportunity. This paper introduces Federated Multi-modal Knowledge Transfer (FedMMKT), a novel framework that enables co-enhancement of a server T2I model and client task-specific models using decentralized multimodal data without compromising data privacy.
Paper Structure (41 sections, 41 equations, 3 figures, 11 tables, 1 algorithm)

This paper contains 41 sections, 41 equations, 3 figures, 11 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Works
  3. Multimodal Federated Learning
  4. Fine-tuning T2I Models with Domain-Specific Data
  5. Federated Fine-tuning of Pre-trained Models
  6. FedMMKT Framework
  7. Problem Statement
  8. Overview
  9. Cross-Modal Data Synthesis
  10. Cross-Client Semantic Alignment
  11. LabVote
  12. MultiRepFusion
  13. Client-Server Knowledge Transfer
  14. Server T2I Fine-Tuning
  15. Client Model Re-Training
  16. ...and 26 more sections

Figures (3)

  • Figure 1: The workflow of the FedMMKT. Different colors represent different data modalities, while varying shades of each color indicate data originating from different clients; dash-dotted and solid borders denote updating and frozen components, respectively.The T2I model contains an encoder, a synthetic latent space, and a decoder.
  • Figure 2: Workflow of MultiRepFusion.
  • Figure 3: Some of the images generated by the T2I model with label bias and LabVote corrected label visualisation