Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

LoRA-Switch: Boosting the Efficiency of Dynamic LLM Adapters via System-Algorithm Co-design

Rui Kong, Qiyang Li, Xinyu Fang, Qingtian Feng, Qingfeng He, Yazhu Dong, Weijun Wang, Yuanchun Li, Linghe Kong, Yunxin Liu

TL;DR

LoRA-Switch identifies that the large latency of dynamic adapters stems from fragmented CUDA kernel calls rather than raw computational complexity. It introduces a token-wise routing scheme and a fused SGMM kernel to pre-merge activated adapters into the backbone before decoding, achieving large latency reductions while preserving accuracy comparable to existing dynamic adapters. The approach demonstrates strong general and domain-specific performance improvements, with decoding latency reductions exceeding 2.4x and modest memory overhead, supported by extensive ablations and comparisons. This system-algorithm co-design offers a practical pathway to deploy flexible, dynamic adapters at scale in LLM serving environments.

Abstract

Recent literature has found that an effective method to customize or further improve large language models (LLMs) is to add dynamic adapters, such as low-rank adapters (LoRA) with Mixture-of-Experts (MoE) structures. Though such dynamic adapters incur modest computational complexity, they surprisingly lead to huge inference latency overhead, slowing down the decoding speed by 2.5+ times. In this paper, we analyze the fine-grained costs of the dynamic adapters and find that the fragmented CUDA kernel calls are the root cause. Therefore, we propose LoRA-Switch, a system-algorithm co-designed architecture for efficient dynamic adapters. Unlike most existing dynamic structures that adopt layer-wise or block-wise dynamic routing, LoRA-Switch introduces a token-wise routing mechanism. It switches the LoRA adapters and weights for each token and merges them into the backbone for inference. For efficiency, this switching is implemented with an optimized CUDA kernel, which fuses the merging operations for all LoRA adapters at once. Based on experiments with popular open-source LLMs on common benchmarks, our approach has demonstrated similar accuracy improvement as existing dynamic adapters, while reducing the decoding latency by more than 2.4 times.

LoRA-Switch: Boosting the Efficiency of Dynamic LLM Adapters via System-Algorithm Co-design

TL;DR

LoRA-Switch identifies that the large latency of dynamic adapters stems from fragmented CUDA kernel calls rather than raw computational complexity. It introduces a token-wise routing scheme and a fused SGMM kernel to pre-merge activated adapters into the backbone before decoding, achieving large latency reductions while preserving accuracy comparable to existing dynamic adapters. The approach demonstrates strong general and domain-specific performance improvements, with decoding latency reductions exceeding 2.4x and modest memory overhead, supported by extensive ablations and comparisons. This system-algorithm co-design offers a practical pathway to deploy flexible, dynamic adapters at scale in LLM serving environments.

Abstract

Recent literature has found that an effective method to customize or further improve large language models (LLMs) is to add dynamic adapters, such as low-rank adapters (LoRA) with Mixture-of-Experts (MoE) structures. Though such dynamic adapters incur modest computational complexity, they surprisingly lead to huge inference latency overhead, slowing down the decoding speed by 2.5+ times. In this paper, we analyze the fine-grained costs of the dynamic adapters and find that the fragmented CUDA kernel calls are the root cause. Therefore, we propose LoRA-Switch, a system-algorithm co-designed architecture for efficient dynamic adapters. Unlike most existing dynamic structures that adopt layer-wise or block-wise dynamic routing, LoRA-Switch introduces a token-wise routing mechanism. It switches the LoRA adapters and weights for each token and merges them into the backbone for inference. For efficiency, this switching is implemented with an optimized CUDA kernel, which fuses the merging operations for all LoRA adapters at once. Based on experiments with popular open-source LLMs on common benchmarks, our approach has demonstrated similar accuracy improvement as existing dynamic adapters, while reducing the decoding latency by more than 2.4 times.
Paper Structure (23 sections, 11 equations, 3 figures, 7 tables, 1 algorithm)

This paper contains 23 sections, 11 equations, 3 figures, 7 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Background and Motivation
  3. Dynamic Adapters
  4. Unexpected Latency Overhead of Dynamic Adapters
  5. Challenge of Reducing Latency Overhead of Dynamic Adapters
  6. Design of LoRA-Switch
  7. Overview
  8. Model Structure
  9. Fused Adapter Switching
  10. SGMM Kernel
  11. Evaluation
  12. Experiment Setup
  13. Accuracy Comparison
  14. Improving General Capability of LLM
  15. Domain-Specific Customization
  16. ...and 8 more sections

Figures (3)

  • Figure 1: Decoding phase execution time profiling of one dynamic adapters layer in MoRAL yang2024moral. Note: The execution time results were preceded by a warm-up of 100 executions and are obtained on the average of 300 executions.
  • Figure 2: Overview of LoRA-Switch.
  • Figure 3: Latency breakdown of one dynamic adapter layer under different settings.