Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

FedMKT: Federated Mutual Knowledge Transfer for Large and Small Language Models

Tao Fan, Guoqiang Ma, Yan Kang, Hanlin Gu, Yuanfeng Song, Lixin Fan, Kai Chen, Qiang Yang

TL;DR

FedMKT tackles the problem of mutually enhancing server large language models (LLMs) and client small language models (SLMs) in a federated setting. It introduces two key components: Bidirectional Token Alignment using a minimum edit distance-based vocabulary mapping and a DualMinCE-driven selective knowledge transfer framework that leverages LoRA adapters and distillation losses $\\mathcal{L}_2$ and $\\mathcal{L}_3$ to transfer knowledge in both directions via a public dataset $\\mathcal{D}_p$. Empirical results across heterogeneous, homogeneous, and one-to-one configurations on multiple public LLMs/SLMs and 6 QA plus 2 instruction-following tasks demonstrate that FedMKT consistently boosts SLM performance and brings the server LLM close to Centralized fine-tuning performance, while preserving data privacy. This approach offers a scalable, privacy-conscious pathway for cross-model knowledge sharing in real-world deployments where clients have limited resources and data heterogeneity.

Abstract

Recent research in federated large language models (LLMs) has primarily focused on enabling clients to fine-tune their locally deployed homogeneous LLMs collaboratively or on transferring knowledge from server-based LLMs to small language models (SLMs) at downstream clients. However, a significant gap remains in the simultaneous mutual enhancement of both the server's LLM and clients' SLMs. To bridge this gap, we propose FedMKT, a parameter-efficient federated mutual knowledge transfer framework for large and small language models. This framework is designed to adaptively transfer knowledge from the server's LLM to clients' SLMs while concurrently enriching the LLM with clients' unique domain insights. We facilitate token alignment using minimum edit distance (MinED) and then selective mutual knowledge transfer between client-side SLMs and a server-side LLM, aiming to collectively enhance their performance. Through extensive experiments across three distinct scenarios, we evaluate the effectiveness of FedMKT using various public LLMs and SLMs on a range of NLP text generation tasks. Empirical results demonstrate that FedMKT simultaneously boosts the performance of both LLMs and SLMs.

FedMKT: Federated Mutual Knowledge Transfer for Large and Small Language Models

TL;DR

FedMKT tackles the problem of mutually enhancing server large language models (LLMs) and client small language models (SLMs) in a federated setting. It introduces two key components: Bidirectional Token Alignment using a minimum edit distance-based vocabulary mapping and a DualMinCE-driven selective knowledge transfer framework that leverages LoRA adapters and distillation losses and to transfer knowledge in both directions via a public dataset . Empirical results across heterogeneous, homogeneous, and one-to-one configurations on multiple public LLMs/SLMs and 6 QA plus 2 instruction-following tasks demonstrate that FedMKT consistently boosts SLM performance and brings the server LLM close to Centralized fine-tuning performance, while preserving data privacy. This approach offers a scalable, privacy-conscious pathway for cross-model knowledge sharing in real-world deployments where clients have limited resources and data heterogeneity.

Abstract

Recent research in federated large language models (LLMs) has primarily focused on enabling clients to fine-tune their locally deployed homogeneous LLMs collaboratively or on transferring knowledge from server-based LLMs to small language models (SLMs) at downstream clients. However, a significant gap remains in the simultaneous mutual enhancement of both the server's LLM and clients' SLMs. To bridge this gap, we propose FedMKT, a parameter-efficient federated mutual knowledge transfer framework for large and small language models. This framework is designed to adaptively transfer knowledge from the server's LLM to clients' SLMs while concurrently enriching the LLM with clients' unique domain insights. We facilitate token alignment using minimum edit distance (MinED) and then selective mutual knowledge transfer between client-side SLMs and a server-side LLM, aiming to collectively enhance their performance. Through extensive experiments across three distinct scenarios, we evaluate the effectiveness of FedMKT using various public LLMs and SLMs on a range of NLP text generation tasks. Empirical results demonstrate that FedMKT simultaneously boosts the performance of both LLMs and SLMs.
Paper Structure (21 sections, 7 equations, 2 figures, 4 tables, 2 algorithms)

This paper contains 21 sections, 7 equations, 2 figures, 4 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Related Work
  3. Model Heterogeneous Federated Learning
  4. Federated Learning for LLMs
  5. The Proposed FedMKT Method
  6. Problem Definition
  7. Bidirectional Token Alignment
  8. Selective Mutual Knowledge Transfer Between LLM and SLMs
  9. Experiments
  10. Setup
  11. Evaluation on Heterogeneous Setting
  12. Evaluation on Homogeneous Setting
  13. Evaluation on One-to-One Setting
  14. Conclusions
  15. FedMKT Workflow
  16. ...and 6 more sections

Figures (2)

  • Figure 1: Overview of the proposed FedMKT workflow. Each communication round of FedMKT involves 11 steps to fine-tune the server' LLM and clients' SLMs.
  • Figure 2: An example for token alignment via MinED.