Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Pilot: Building the Federated Multimodal Instruction Tuning Framework

Baochen Xiong, Xiaoshan Yang, Yaguang Song, Yaowei Wang, Changsheng Xu

TL;DR

This work defines Federated Multimodal Instruction Tuning (FedMIT) and presents Pilot, a two-stage adapter-on-adapter framework with Cross-task Mixture-of-Adapters (CT-MoA) to enable cross-task knowledge transfer among heterogeneous multimodal clients. Pilot uses task- and client-specific adapters in Stage 1 to extract personalized features, then forms cross-task interactions in Stage 2 via CT-MoA with a router and auxiliary losses, coupled with an adaptive text-adapter aggregation strategy. Empirical results on two cross-task federated scenarios with LLaVA show that Pilot consistently outperforms FedAvg, Shepherd, and FedDPA, demonstrating robustness to task heterogeneity and privacy-preserving collaboration. The approach offers a practical path for privacy-preserving, cross-task multimodal instruction tuning with efficient parameter sharing and minimal centralized data reliance.

Abstract

In this paper, we explore a novel federated multimodal instruction tuning task(FedMIT), which is significant for collaboratively fine-tuning MLLMs on different types of multimodal instruction data on distributed devices. To solve the new task, we propose a federated multimodal instruction tuning framework(Pilot). Our framework integrates two stages of "adapter on adapter" into the connector of the vision encoder and the LLM. In stage 1, we extract task-specific features and client-specific features from visual information. In stage 2, we build the cross-task Mixture-of-Adapters(CT-MoA) module to perform cross-task interaction. Each client can not only capture personalized information of local data and learn task-related multimodal information, but also learn general knowledge from other tasks. In addition, we introduce an adaptive parameter aggregation strategy for text training parameters, which optimizes parameter aggregation by calculating weights based on the euclidean distance between parameters, so that parameter aggregation can benefit from positive effects to the greatest extent while effectively reducing negative effects. Our framework can collaboratively exploit distributed data from different local clients to learn cross-task knowledge without being affected by the task heterogeneity during instruction tuning. The effectiveness of our method is verified in two different cross-task scenarios.

Pilot: Building the Federated Multimodal Instruction Tuning Framework

TL;DR

This work defines Federated Multimodal Instruction Tuning (FedMIT) and presents Pilot, a two-stage adapter-on-adapter framework with Cross-task Mixture-of-Adapters (CT-MoA) to enable cross-task knowledge transfer among heterogeneous multimodal clients. Pilot uses task- and client-specific adapters in Stage 1 to extract personalized features, then forms cross-task interactions in Stage 2 via CT-MoA with a router and auxiliary losses, coupled with an adaptive text-adapter aggregation strategy. Empirical results on two cross-task federated scenarios with LLaVA show that Pilot consistently outperforms FedAvg, Shepherd, and FedDPA, demonstrating robustness to task heterogeneity and privacy-preserving collaboration. The approach offers a practical path for privacy-preserving, cross-task multimodal instruction tuning with efficient parameter sharing and minimal centralized data reliance.

Abstract

In this paper, we explore a novel federated multimodal instruction tuning task(FedMIT), which is significant for collaboratively fine-tuning MLLMs on different types of multimodal instruction data on distributed devices. To solve the new task, we propose a federated multimodal instruction tuning framework(Pilot). Our framework integrates two stages of "adapter on adapter" into the connector of the vision encoder and the LLM. In stage 1, we extract task-specific features and client-specific features from visual information. In stage 2, we build the cross-task Mixture-of-Adapters(CT-MoA) module to perform cross-task interaction. Each client can not only capture personalized information of local data and learn task-related multimodal information, but also learn general knowledge from other tasks. In addition, we introduce an adaptive parameter aggregation strategy for text training parameters, which optimizes parameter aggregation by calculating weights based on the euclidean distance between parameters, so that parameter aggregation can benefit from positive effects to the greatest extent while effectively reducing negative effects. Our framework can collaboratively exploit distributed data from different local clients to learn cross-task knowledge without being affected by the task heterogeneity during instruction tuning. The effectiveness of our method is verified in two different cross-task scenarios.
Paper Structure (18 sections, 12 equations, 3 figures, 6 tables)

This paper contains 18 sections, 12 equations, 3 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Federated Learning
  4. Multimodal Instruction Tuning
  5. Federated Instruction Tuning
  6. Methodology
  7. Problem Definition
  8. Federated Multimodal Instruction Tuning Framework
  9. Federated Optimization
  10. Experiment
  11. Experimental Setups
  12. Baselines
  13. Implementation Details
  14. Experimental Results
  15. Ablation Studies
  16. ...and 3 more sections

Figures (3)

  • Figure 1: (a) Comparison of the FedIT task with the FedMIT task. (b) Compare the relative scores of traditional FedIT methods on the two tasks.
  • Figure 2: Overview of the proposed Federated Multimodal Instruction Tuning Framework (Pilot).
  • Figure 3: Cross-task Mixture-of-Adapters (CT-MoA).