Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

A Step Toward Federated Pretraining of Multimodal Large Language Models

Baochen Xiong, Yifan Xu, Xiaoshan Yang, Yaguang Song, Yaowei Wang, Changsheng Xu

Abstract

The rapid evolution of Multimodal Large Language Models (MLLMs) is bottlenecked by the saturation of high-quality public data, while vast amounts of diverse multimodal data remain inaccessible in privacy-sensitive silos. Federated Learning (FL) offers a promising solution to unlock these distributed resources, but existing research focuses predominantly on fine-tuning, leaving the foundational pre-training phase largely unexplored. In this paper, we formally introduce the Federated MLLM Alignment (Fed-MA) task, a lightweight pre-training paradigm that freezes the vision encoder and LLM while collaboratively training the cross-modal projector. We identify two critical challenges in this setting: (i) parameter interference in aggregating local projectors; and (ii) gradient oscillations in one-pass collaborative SGD. To address these challenges, we propose Fed-CMP, a pioneering framework for federated MLLM pre-training. Fed-CMP employs Canonical Reliability-Aware Aggregation, which constructs a canonical space to decompose client projectors into a shared alignment basis and client-specific coefficients, then performs reliability-weighted fusion to suppress parameter interference. Furthermore, Fed-CMP introduces Orthogonality-Preserved Momentum, which applies momentum to the shared alignment basis via orthogonal projection, accumulating historical optimization directions while preserving geometric structure. We construct four federated pre-training scenarios based on public datasets, and extensive experiments validate that Fed-CMP significantly outperforms existing baselines.

A Step Toward Federated Pretraining of Multimodal Large Language Models

Abstract

The rapid evolution of Multimodal Large Language Models (MLLMs) is bottlenecked by the saturation of high-quality public data, while vast amounts of diverse multimodal data remain inaccessible in privacy-sensitive silos. Federated Learning (FL) offers a promising solution to unlock these distributed resources, but existing research focuses predominantly on fine-tuning, leaving the foundational pre-training phase largely unexplored. In this paper, we formally introduce the Federated MLLM Alignment (Fed-MA) task, a lightweight pre-training paradigm that freezes the vision encoder and LLM while collaboratively training the cross-modal projector. We identify two critical challenges in this setting: (i) parameter interference in aggregating local projectors; and (ii) gradient oscillations in one-pass collaborative SGD. To address these challenges, we propose Fed-CMP, a pioneering framework for federated MLLM pre-training. Fed-CMP employs Canonical Reliability-Aware Aggregation, which constructs a canonical space to decompose client projectors into a shared alignment basis and client-specific coefficients, then performs reliability-weighted fusion to suppress parameter interference. Furthermore, Fed-CMP introduces Orthogonality-Preserved Momentum, which applies momentum to the shared alignment basis via orthogonal projection, accumulating historical optimization directions while preserving geometric structure. We construct four federated pre-training scenarios based on public datasets, and extensive experiments validate that Fed-CMP significantly outperforms existing baselines.

Paper Structure

This paper contains 30 sections, 19 equations, 9 figures, 9 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Multimodal Large Language Models
  4. Federated Learning
  5. Methodology
  6. Task Definition
  7. Federated MLLM Pre-training Framework
  8. Canonical Reliability-Aware Aggregation
  9. Orthogonality-Preserved Momentum
  10. Experiments
  11. Experimental Setups
  12. Experimental Results
  13. Ablation Studies
  14. Discussion
  15. Analysis of Reliability-Weighted Fusion
  16. ...and 15 more sections

Figures (9)

  • Figure 1: A step toward Federated MLLMs: Comparison among conventional Federated Learning, centralized MLLMs training, and the proposed Federated MLLMs Alignment (Fed-MA) task.
  • Figure 2: Overview of the proposed Federated MLLM pretraining framework (Fed-CMP).
  • Figure 3: t-SNE visualization of heterogeneous clustering on the CC12M dataset, utilizing 5 distinct clusters across 4 similarity metrics.
  • Figure 4: Performance scores across communication rounds under the $\mathcal{M}_{I-I}$ setting.
  • Figure 5: t-SNE visualization of clustering results using four distinct similarity metrics.
  • ...and 4 more figures