MoPEFT: A Mixture-of-PEFTs for the Segment Anything Model

TL;DR

Problem: Fine-tuning large foundation models is costly, and different PEFTs impact different parts of the transformer, complicating method choice. Approach: Proposes MoPEFT, a Mixture-of-PEFTs framework that nests LoRA, Prefix Tuning, and Adapters as submodules and uses a gating mechanism to select among them based on data-task. Contributions: (i) survey of PEFT methods across domains, (ii) MoPEFT design with MoE-inspired gates, (iii) empirical evidence showing improved performance on MESS over individual PEFTs. Impact: enables efficient, task-adaptive fine-tuning for segmentation across diverse domains with limited additional parameters, potentially generalizable to other foundation-model fine-tuning scenarios.

Abstract

The emergence of foundation models, such as the Segment Anything Model (SAM), has sparked interest in Parameter-Efficient Fine-Tuning (PEFT) methods that tailor these large models to application domains outside their training data. However, different PEFT techniques modify the representation of a model differently, making it a non-trivial task to select the most appropriate method for the domain of interest. We propose a new framework, Mixture-of-PEFTs methods (MoPEFT), that is inspired by traditional Mixture-of-Experts (MoE) methodologies and is utilized for fine-tuning SAM. Our MoPEFT framework incorporates three different PEFT techniques as submodules and dynamically learns to activate the ones that are best suited for a given data-task setup. We test our method on the Segment Anything Model and show that MoPEFT consistently outperforms other fine-tuning methods on the MESS benchmark.

Figures (2)

• Figure 1: An overview of our MoPEFTs framework
• Figure 2: Number of times each PEFT method is called during inference. Different datasets display distinct patterns