FedUMM: A General Framework for Federated Learning with Unified Multimodal Models

TL;DR

FedUMM presents a general federated learning framework for Unified Multimodal Models that enables privacy-preserving collaboration across distributed clients by freezing a large backbone and training lightweight LoRA adapters. It introduces device-edge partitioning and a semantic-aware fusion (FedUMM-Fusion) to handle cross-modal heterogeneity with an efficient aggregation rule $\theta_{\text{global}} = \sum_{k=1}^{K} w_k \cdot \theta_k$ where $w_k = \frac{n_k}{\sum_{j=1}^{K} n_j} \cdot \alpha_k$. Experiments on VQA v2 and GenEval show FedUMM achieves near-centralized performance while drastically reducing communication costs (over 99% per-round reduction) and maintaining robustness across up to 16 clients under Dirichlet-driven non-IID distributions. The work delivers a practical baseline for privacy-preserving federated training of large unified multimodal models and highlights the benefits of modality-aware adapters and semantic-aware aggregation for real-world deployment.

Abstract

Unified multimodal models (UMMs) are emerging as strong foundation models that can do both generation and understanding tasks in a single architecture. However, they are typically trained in centralized settings where all training and downstream datasets are gathered in a central server, limiting the deployment in privacy-sensitive and geographically distributed scenarios. In this paper, we present FedUMM, a general federated learning framework for UMMs under non-IID multimodal data with low communication cost. Built on NVIDIA FLARE, FedUMM instantiates federation for a BLIP3o backbone via parameter-efficient fine-tuning: clients train lightweight LoRA adapters while freezing the foundation models, and the server aggregates only adapter updates. We evaluate on VQA v2 and the GenEval compositional generation benchmarks under Dirichlet-controlled heterogeneity with up to 16 clients. Results show slight degradation as client count and heterogeneity increase, while remaining competitive with centralized training. We further analyze computation--communication trade-offs and demonstrate that adapter-only federation reduces per-round communication by over an order of magnitude compared to full fine-tuning, enabling practical federated UMM training. This work provides empirical experience for future research on privacy-preserving federated unified multimodal models.

Figures (4)

• Figure 1: Motivation of FedUMM. Existing UMMs are trained exclusively on public datasets in centralized settings, while vast amounts of valuable private multimodal data from hospitals, banks, and enterprises remain inaccessible due to privacy constraints.
• Figure 2: Motivation of FedUMM. Deploying unified multimodal models in federated settings faces three key challenges: (1) Data heterogeneity: different clients have different modality combinations (e.g., vision, audio, text); (2) Privacy constraints: sensitive data cannot be uploaded for centralized training; (3) Architecture complexity: UMMs like BLIP3o contain billions of parameters across multiple modules, leading to high communication costs and training difficulties.
• Figure 3: Overview of FedUMM. LoRA adapters are plugged into frozen BLIP3o modules and aggregated across clients via federated learning.
• Figure 4: Qualitative results of FedUMM on unified multimodal understanding and generation.Top row (T2I): Text-to-image generation results. Given diverse text prompts ranging from natural scenes (sunset, enchanted forest) to complex compositions (steampunk owl, cat reading), our federated model generates high-quality, semantically aligned images that faithfully capture the described content, style, and atmosphere. Bottom row (I2T): Image-to-text understanding results. Given input images, the model generates accurate and detailed captions that correctly identify objects, actions, spatial relationships, and scene context. These results demonstrate that FedUMM preserves the unified understanding-generation capabilities of BLIP3-o while enabling privacy-preserving federated training across distributed clients.