Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

pMoE: Prompting Diverse Experts Together Wins More in Visual Adaptation

Shentong Mo, Xufang Luo, Dongsheng Li

TL;DR

This work proposes a novel Mixture-of-Experts prompt tuning method called pMoE, which leverages the strengths of multiple expert domains through expert-specialized prompt tokens and the learnable dispatcher, effectively combining their expertise in a unified model framework.

Abstract

Parameter-efficient fine-tuning has demonstrated promising results across various visual adaptation tasks, such as classification and segmentation. Typically, prompt tuning techniques have harnessed knowledge from a single pre-trained model, whether from a general or a specialized medical domain. However, this approach typically overlooks the potential synergies that could arise from integrating diverse domain knowledge within the same tuning process. In this work, we propose a novel Mixture-of-Experts prompt tuning method called pMoE, which leverages the strengths of multiple expert domains through expert-specialized prompt tokens and the learnable dispatcher, effectively combining their expertise in a unified model framework. Our pMoE introduces expert-specific prompt tokens and utilizes a dynamic token dispatching mechanism at various prompt layers to optimize the contribution of each domain expert during the adaptation phase. By incorporating both domain knowledge from diverse experts, the proposed pMoE significantly enhances the model's versatility and applicability to a broad spectrum of tasks. We conduct extensive experiments across 47 adaptation tasks, including both classification and segmentation in general and medical domains. The results demonstrate that our pMoE not only achieves superior performance with a large margin of improvements but also offers an optimal trade-off between computational efficiency and adaptation effectiveness compared to existing methods.

pMoE: Prompting Diverse Experts Together Wins More in Visual Adaptation

TL;DR

This work proposes a novel Mixture-of-Experts prompt tuning method called pMoE, which leverages the strengths of multiple expert domains through expert-specialized prompt tokens and the learnable dispatcher, effectively combining their expertise in a unified model framework.

Abstract

Parameter-efficient fine-tuning has demonstrated promising results across various visual adaptation tasks, such as classification and segmentation. Typically, prompt tuning techniques have harnessed knowledge from a single pre-trained model, whether from a general or a specialized medical domain. However, this approach typically overlooks the potential synergies that could arise from integrating diverse domain knowledge within the same tuning process. In this work, we propose a novel Mixture-of-Experts prompt tuning method called pMoE, which leverages the strengths of multiple expert domains through expert-specialized prompt tokens and the learnable dispatcher, effectively combining their expertise in a unified model framework. Our pMoE introduces expert-specific prompt tokens and utilizes a dynamic token dispatching mechanism at various prompt layers to optimize the contribution of each domain expert during the adaptation phase. By incorporating both domain knowledge from diverse experts, the proposed pMoE significantly enhances the model's versatility and applicability to a broad spectrum of tasks. We conduct extensive experiments across 47 adaptation tasks, including both classification and segmentation in general and medical domains. The results demonstrate that our pMoE not only achieves superior performance with a large margin of improvements but also offers an optimal trade-off between computational efficiency and adaptation effectiveness compared to existing methods.
Paper Structure (14 sections, 5 equations, 2 figures, 13 tables, 1 algorithm)

This paper contains 14 sections, 5 equations, 2 figures, 13 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Method
  4. Visual Prompt Tuning for a Single Model
  5. Expert Prompt Tokens
  6. Dynamically Select and Fuse Tokens with the Dispatcher
  7. Experiments
  8. Experimental setup
  9. Comparison to prior work
  10. Experimental Analysis
  11. Conclusion
  12. Experimental Details
  13. Algorithm for pMoE
  14. Experimental Analysis

Figures (2)

  • Figure 1: Illustration of the proposed pMoE framework. Here, we demonstrate how the dispatcher handles tokens and produces integrated prompt tokens for Expert $k$, with the same method applied to other experts as well. The dynamic dispatching method takes expert prompt tokens from all experts and the state of the current expert as inputs, and outputs dispatching weights for controlling portions to integrate prompt tokens for the next layer. Different colors represent distinct weight groups, applied to corresponding expert prompt tokens, yielding different integrated prompt tokens. This dynamic dispatching mechanism ensures communication and interaction among diverse experts, making the model contribute the most relevant knowledge to the final output.
  • Figure 2: Visualization of Mixture-of-Experts path. Our pMoE can dynamically choose a distinct, task-specific path of experts for each benchmark type, demonstrating the ability to adapt to the particularities of each task.