Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Monkey Jump : MoE-Style PEFT for Efficient Multi-Task Learning

Nusrat Jahan Prottasha, Md Kowsher, Chun-Nam Yu, Chen Chen, Ozlem Garibay

TL;DR

Monkey Jump introduces gradient-free routing to parameter-efficient fine-tuning by treating existing transformer adapters as implicit experts and routing tokens via $k$-means centers updated with EMA. This preserves the fixed parameter budget of standard PEFT while achieving MoE-style specialization, avoiding the trainable routers and multi-expert activation costs of traditional MoE-PEFT. Theoretical results show token-wise routing increases expressivity and last-token representations maximize information for sequence-wise routing in causal transformers. Empirically, MJ matches or surpasses MoE-PEFT performance across 47 benchmarks spanning text, image, and video, with 7–29× fewer trainable parameters, up to 48% lower memory, and 1.5–2× faster training, demonstrating broad applicability and practical efficiency gains.

Abstract

Mixture-of-experts variants of parameter-efficient fine-tuning enable per-token specialization, but they introduce additional trainable routers and expert parameters, increasing memory usage and training cost. This undermines the core goal of parameter-efficient fine-tuning. We propose Monkey Jump, a method that brings mixture-of-experts-style specialization to parameter-efficient fine-tuning without introducing extra trainable parameters for experts or routers. Instead of adding new adapters as experts, Monkey Jump treats the adapters already present in each Transformer block (such as query, key, value, up, and down projections) as implicit experts and routes tokens among them. Routing is performed using k-means clustering with exponentially moving averaged cluster centers, requiring no gradients and no learned parameters. We theoretically show that token-wise routing increases expressivity and can outperform shared adapters by avoiding cancellation effects. Across multi-task experiments covering 14 text, 14 image, and 19 video benchmarks, Monkey Jump achieves competitive performance with mixture-of-experts-based parameter-efficient fine-tuning methods while using 7 to 29 times fewer trainable parameters, up to 48 percent lower memory consumption, and 1.5 to 2 times faster training. Monkey Jump is architecture-agnostic and can be applied to any adapter-based parameter-efficient fine-tuning method.

Monkey Jump : MoE-Style PEFT for Efficient Multi-Task Learning

TL;DR

Monkey Jump introduces gradient-free routing to parameter-efficient fine-tuning by treating existing transformer adapters as implicit experts and routing tokens via -means centers updated with EMA. This preserves the fixed parameter budget of standard PEFT while achieving MoE-style specialization, avoiding the trainable routers and multi-expert activation costs of traditional MoE-PEFT. Theoretical results show token-wise routing increases expressivity and last-token representations maximize information for sequence-wise routing in causal transformers. Empirically, MJ matches or surpasses MoE-PEFT performance across 47 benchmarks spanning text, image, and video, with 7–29× fewer trainable parameters, up to 48% lower memory, and 1.5–2× faster training, demonstrating broad applicability and practical efficiency gains.

Abstract

Mixture-of-experts variants of parameter-efficient fine-tuning enable per-token specialization, but they introduce additional trainable routers and expert parameters, increasing memory usage and training cost. This undermines the core goal of parameter-efficient fine-tuning. We propose Monkey Jump, a method that brings mixture-of-experts-style specialization to parameter-efficient fine-tuning without introducing extra trainable parameters for experts or routers. Instead of adding new adapters as experts, Monkey Jump treats the adapters already present in each Transformer block (such as query, key, value, up, and down projections) as implicit experts and routes tokens among them. Routing is performed using k-means clustering with exponentially moving averaged cluster centers, requiring no gradients and no learned parameters. We theoretically show that token-wise routing increases expressivity and can outperform shared adapters by avoiding cancellation effects. Across multi-task experiments covering 14 text, 14 image, and 19 video benchmarks, Monkey Jump achieves competitive performance with mixture-of-experts-based parameter-efficient fine-tuning methods while using 7 to 29 times fewer trainable parameters, up to 48 percent lower memory consumption, and 1.5 to 2 times faster training. Monkey Jump is architecture-agnostic and can be applied to any adapter-based parameter-efficient fine-tuning method.
Paper Structure (89 sections, 4 theorems, 107 equations, 12 figures, 13 tables)

This paper contains 89 sections, 4 theorems, 107 equations, 12 figures, 13 tables.

Table of Contents

  1. Introduction
  2. Contributions.
  3. Preliminaries
  4. PEFT.
  5. MoE.
  6. MoE-PEFT.
  7. Notation.
  8. Monkey Jump
  9. Routing
  10. Center Initialization and Online Updates
  11. Initialization.
  12. Online updates.
  13. Routing Variants
  14. Token-wise routing.
  15. Sequence-wise routing.
  16. ...and 74 more sections

Key Result

Theorem 1

Under hard routing, if all adapters are activated and receive sufficiently diverse inputs (i.e., $\mathrm{rank}(\Delta W_e H_e) = \mathrm{rank}(\Delta W_e)$ for all $e$), then with strict inequality whenever $\mathrm{Col}(\sum_e \Delta W_e) \subsetneq \sum_e \mathcal{C}_e$.

Figures (12)

  • Figure 1: Overview of MJ compared to MoE-PEFT. (a) MoE-PEFT architecture: each projection (Q, K, V, O, Gate, Up, Down) has multiple expert adapters with a learned router. (b) MoE-PEFT routing: a trainable router selects among $N$ expert adapters, and outputs are summed ($\Sigma$). (c) MJ architecture: each projection has a single adapter (same as standard PEFT). Here, V and Down adapters are activated ($k{=}2$); the rest are skipped. Inactive projections apply only frozen weights $\textcolor{lightblue}{\mathbf{W_e}}$; active projections apply $\textcolor{lightblue}{\mathbf{W_e}} + m_e \cdot \textcolor{orange!85!black}{\mathbf{\Delta W_e}}$. (d) MJ routing mechanism: ① Initialize cluster centers $\textcolor{teal}{\mathbf{C}}$ via $k$-means before training; ② For input token $h_t$, ③ compute cosine similarity to each center; ④ Select top-$k$ experts based on similarity (here $e_2$ is activated, $e_1$ and $e_3$ are skipped).
  • Figure 2: Efficiency comparison. Top row: Trainable parameters (K), total parameters (M), model size (MB), and peak GPU memory (GB). Bottom row: Training throughput (it/s = iterations per second), training time (min), and inference throughput across GLUE tasks.
  • Figure 3: (a) $K$-means initialization matches trainable routers. (b) More samples improve initialization, saturating at 5K--10K. (c) EMA update coverage of 50--70% suffices. (d) More routing layers improve performance.
  • Figure 4: Routing hyper-parameters. (a) Similarity function: cosine similarity outperforms distance-based metrics. (b) Routing temperature $\tau$: best at 1.0, stable across [0.5, 1.5]. (c) EMA smoothing $\beta$: best at 0.7, stable across [0.5, 0.9]. (d) Update schedule: performance improves with updates until 1500--2000 steps, then saturates.
  • Figure 5: Expert usage across layers and GLUE tasks. Each heatmap shows the percentage of tokens routed to each attention projection (Q, K, V, O). Different tasks prefer different projections, and preferences evolve across layers—validating the submodules-as-experts design.
  • ...and 7 more figures

Theorems & Definitions (12)

  • Theorem 1: Expressivity of MJ
  • Theorem 2: Information Maximality
  • proof : Proof sketch
  • Proposition 1: Soft Routing Expressivity
  • Definition 1: Entropy
  • Definition 2: Conditional Entropy
  • Definition 3: Mutual Information
  • Definition 4: Conditional Mutual Information
  • Definition 5: Kullback-Leibler Divergence
  • Definition 6: Causal Hidden Representation
  • ...and 2 more