Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

MuxTune: Efficient Multi-Task LLM Fine-Tuning in Multi-Tenant Datacenters via Spatial-Temporal Backbone Multiplexing

Chunyu Xue, Yi Pan, Weihao Cui, Quan Chen, Shulai Zhang, Bingsheng He, Minyi Guo

TL;DR

MuxTune proposes hierarchical co-scheduling scheme with task, operator, and data-level optimizations that enables resource-efficient concurrent execution of multiple PEFT tasks and employs chunk-based data alignment to mitigate inter-task ineffective tokens.

Abstract

Parameter-Efficient Fine-Tuning (PEFT) is widely applied as the backend of fine-tuning APIs for large language model (LLM) customization in datacenters. Service providers deploy separate instances for individual PEFT tasks, giving rise to prominent resource inefficiencies, including (1) GPU underutilization from small-scale, PEFT-native operators and (2) device stalls from communication delays and data dependencies in parallelized execution. To address these issues, this paper presents MuxTune, a fine-tuning system that enables resource-efficient concurrent execution of multiple PEFT tasks. The key idea is to multiplex the backbone across independent tasks in a spatial-temporal manner for improved utilization and reduced stalls. Building on flexible, modularized backbone sharing via unified PEFT representations, MuxTune proposes hierarchical co-scheduling scheme with task, operator, and data-level optimizations. Specifically, it fuses tasks through a hybrid of spatial and temporal multiplexing, and orchestrates multi-task operator execution in two-tiered hybrid parallelism. Additionally, MuxTune employs chunk-based data alignment to mitigate inter-task ineffective tokens. Experimental results demonstrate that MuxTune achieves up to $2.33\times$ higher throughput and $5.29\times$ memory reduction compared to three state-of-the-art baselines.

MuxTune: Efficient Multi-Task LLM Fine-Tuning in Multi-Tenant Datacenters via Spatial-Temporal Backbone Multiplexing

TL;DR

MuxTune proposes hierarchical co-scheduling scheme with task, operator, and data-level optimizations that enables resource-efficient concurrent execution of multiple PEFT tasks and employs chunk-based data alignment to mitigate inter-task ineffective tokens.

Abstract

Parameter-Efficient Fine-Tuning (PEFT) is widely applied as the backend of fine-tuning APIs for large language model (LLM) customization in datacenters. Service providers deploy separate instances for individual PEFT tasks, giving rise to prominent resource inefficiencies, including (1) GPU underutilization from small-scale, PEFT-native operators and (2) device stalls from communication delays and data dependencies in parallelized execution. To address these issues, this paper presents MuxTune, a fine-tuning system that enables resource-efficient concurrent execution of multiple PEFT tasks. The key idea is to multiplex the backbone across independent tasks in a spatial-temporal manner for improved utilization and reduced stalls. Building on flexible, modularized backbone sharing via unified PEFT representations, MuxTune proposes hierarchical co-scheduling scheme with task, operator, and data-level optimizations. Specifically, it fuses tasks through a hybrid of spatial and temporal multiplexing, and orchestrates multi-task operator execution in two-tiered hybrid parallelism. Additionally, MuxTune employs chunk-based data alignment to mitigate inter-task ineffective tokens. Experimental results demonstrate that MuxTune achieves up to higher throughput and memory reduction compared to three state-of-the-art baselines.
Paper Structure (66 sections, 5 theorems, 9 equations, 22 figures, 2 tables, 1 algorithm)

This paper contains 66 sections, 5 theorems, 9 equations, 22 figures, 2 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Background and Motivation
  3. Parameter-Efficient Fine-Tuning (PEFT)
  4. Basics.
  5. Characteristics.
  6. Inefficiencies of PEFT Workloads
  7. Insufficient Utilization.
  8. Device Stall.
  9. Intuitive Multiplexing Approaches
  10. Coarse-grained spatial multiplexing causes poor scalability and limited performance gains.
  11. Batching-based spatial multiplexing causes enlarged stalls and ineffective computation.
  12. Temporal multiplexing and overlapping result in even lower GPU utilization.
  13. Our Approach and Challenges
  14. System Design
  15. Overview
  16. ...and 51 more sections

Key Result

Lemma 1

The end-to-end pipeline latency is calculated by adding the latencies of three phases ($T_{warm}$, $T_{steady}$, $T_{drain}$), while $T_{warm}$/$T_{drain}$ can be calculated as: $(S - 1)t_{1}$/$(S - 1)t_{P}$.

Figures (22)

  • Figure 1: Workflow of submitting tasks and fine-tuning LLMs remotely. Each instance separately handles $1$ task of diverse PEFT types, with its backbone parallelized on $2$ GPUs.
  • Figure 2: Representative categories of PEFT algorithms.
  • Figure 3: PEFT inefficiencies (MBS: micro-batch size, sequence length 128). (a) Single-GPU MFU of $8$-layer models (global batch size $32$, LLA: LLaMA7B, GPT: GPT2.7B). (b) Operator utilization (shape [MBS,128,4096]$\times$[4096,r], r=4096 for pretraining). (c) Multi-GPU MFU of full models (global batch size $128$). (d) GPU and NVLink utilization.
  • Figure 4: Device stalls in PEFT under model parallelism.
  • Figure 5: General views of intuitive multiplexing approaches.
  • ...and 17 more figures

Theorems & Definitions (5)

  • Lemma 1
  • Lemma 2
  • Theorem 1
  • Lemma 3
  • Theorem 2