Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

PT-MoE: An Efficient Finetuning Framework for Integrating Mixture-of-Experts into Prompt Tuning

Zongqian Li, Yixuan Su, Nigel Collier

TL;DR

This paper tackles the limited efficiency and domain sensitivity of existing prompt-tuning methods by proposing PT-MoE, a framework that fuses low-rank matrix decomposition with dynamic mixture-of-experts routing in prompt tuning. By decomposing each soft prompt as $P_i = A_i B$ and routing inputs through a learned hard selector, PT-MoE achieves strong performance across 17 datasets in both QA and mathematical problem solving while using substantially fewer trainable parameters. Across MRQA QA and GSM8K-like math tasks, PT-MoE outperforms standard PT, SMoP, and several LoRA-based baselines, and ablations pinpoint optimal prompt length, expert counts, and routing configurations, providing practical guidance for PEFT design. The findings demonstrate robust cross-task generalization, efficiency gains, and reduced hallucinations, highlighting PT-MoE's potential to drive scalable, task-adaptive fine-tuning in large language models.

Abstract

Parameter-efficient fine-tuning (PEFT) methods have shown promise in adapting large language models, yet existing approaches exhibit counter-intuitive phenomena: integrating router into prompt tuning (PT) increases training efficiency yet does not improve performance universally; parameter reduction through matrix decomposition can improve performance in specific domains. Motivated by these observations and the modular nature of PT, we propose PT-MoE, a novel framework that integrates matrix decomposition with mixture-of-experts (MoE) routing for efficient PT. Results across 17 datasets demonstrate that PT-MoE achieves state-of-the-art performance in both question answering (QA) and mathematical problem solving tasks, improving F1 score by 1.49 points over PT and 2.13 points over LoRA in QA tasks, while enhancing mathematical accuracy by 10.75 points over PT and 0.44 points over LoRA, all while using 25% fewer parameters than LoRA. Our analysis reveals that while PT methods generally excel in QA tasks and LoRA-based methods in math datasets, the integration of matrix decomposition and MoE in PT-MoE yields complementary benefits: decomposition enables efficient parameter sharing across experts while MoE provides dynamic adaptation, collectively enabling PT-MoE to demonstrate cross-task consistency and generalization abilities. These findings, along with ablation studies on routing mechanisms and architectural components, provide insights for future PEFT methods.

PT-MoE: An Efficient Finetuning Framework for Integrating Mixture-of-Experts into Prompt Tuning

TL;DR

This paper tackles the limited efficiency and domain sensitivity of existing prompt-tuning methods by proposing PT-MoE, a framework that fuses low-rank matrix decomposition with dynamic mixture-of-experts routing in prompt tuning. By decomposing each soft prompt as and routing inputs through a learned hard selector, PT-MoE achieves strong performance across 17 datasets in both QA and mathematical problem solving while using substantially fewer trainable parameters. Across MRQA QA and GSM8K-like math tasks, PT-MoE outperforms standard PT, SMoP, and several LoRA-based baselines, and ablations pinpoint optimal prompt length, expert counts, and routing configurations, providing practical guidance for PEFT design. The findings demonstrate robust cross-task generalization, efficiency gains, and reduced hallucinations, highlighting PT-MoE's potential to drive scalable, task-adaptive fine-tuning in large language models.

Abstract

Parameter-efficient fine-tuning (PEFT) methods have shown promise in adapting large language models, yet existing approaches exhibit counter-intuitive phenomena: integrating router into prompt tuning (PT) increases training efficiency yet does not improve performance universally; parameter reduction through matrix decomposition can improve performance in specific domains. Motivated by these observations and the modular nature of PT, we propose PT-MoE, a novel framework that integrates matrix decomposition with mixture-of-experts (MoE) routing for efficient PT. Results across 17 datasets demonstrate that PT-MoE achieves state-of-the-art performance in both question answering (QA) and mathematical problem solving tasks, improving F1 score by 1.49 points over PT and 2.13 points over LoRA in QA tasks, while enhancing mathematical accuracy by 10.75 points over PT and 0.44 points over LoRA, all while using 25% fewer parameters than LoRA. Our analysis reveals that while PT methods generally excel in QA tasks and LoRA-based methods in math datasets, the integration of matrix decomposition and MoE in PT-MoE yields complementary benefits: decomposition enables efficient parameter sharing across experts while MoE provides dynamic adaptation, collectively enabling PT-MoE to demonstrate cross-task consistency and generalization abilities. These findings, along with ablation studies on routing mechanisms and architectural components, provide insights for future PEFT methods.
Paper Structure (18 sections, 4 figures, 9 tables, 1 algorithm)

This paper contains 18 sections, 4 figures, 9 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Methods
  4. Experimental Design
  5. Datasets
  6. Gold Standard and Baselines
  7. Evaluation Metrices
  8. Models
  9. Results
  10. Question Answering
  11. Mathematical Problem Solving
  12. Case Study
  13. Ablation Studies
  14. Efficiency Analysis
  15. Conclusions
  16. ...and 3 more sections

Figures (4)

  • Figure 1: Performance comparison of PEFT methods on 12 QA datasets in the MRQA benchmark (upper) and 5 math datasets (lower). ↑ indicates higher is better; ↓ indicates lower is better.
  • Figure 2: Framework of PT-MoE. Each soft prompt is decomposed into an input-specific matrix $A_i$ and a shared matrix $B$, with a router adaptively selecting and combining prompt components based on input. The resulting soft prompt is prepended to the input for the frozen LLM.
  • Figure 3: Ablation studies on key components of PT-MoE, showing the influence of (Left) prompt length, (Center left) number of experts, (Center right) trainable parameters, and (Right) routing mechanisms ((N)S: (Non-)Selective, (N)P: (Non-)Probationary) on in-domain (ID) and out-of-domain (OOD) performance.
  • Figure 4: Parameter and training efficiency comparison across different methods. The x-axis shows training steps for the highest performance after training parameter search, while the y-axis shows the average accuracy on math datasets. Circle sizes indicate the number of trainable parameters, with larger circles indicating more parameters.