Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

zFLoRA: Zero-Latency Fused Low-Rank Adapters

Dhananjaya Gowda, Seoha Song, Harshith Goka, Junhyun Lee

TL;DR

The paper tackles the latency overhead of adapter-based PEFT methods in LLM inference by introducing zFLoRA, a zero-latency fused low-rank adapter that merges adapter computations with base model projections. It designs a fusion strategy that expands input from $d$ to $d+r$ once and merges back to $d$ only at the LM head, enabling cross-block fusion across MHA and FFN while avoiding per-layer input-expansion and output-merge costs. Empirically, zFLoRA matches the performance of LoRA and full fine-tuning (FFT) across 18 tasks on 1B, 3B, and 7B LLaMA models, while delivering negligible latency on both GPU (NVIDIA H100, via vLLM) and mobile NPU (Galaxy S25+), underscoring its practicality for on-device and edge deployments. The results indicate that zFLoRA provides a practical path to fast, task-specific fine-tuning with minimal compute overhead, enabling broader deployment of adaptable LLMs in latency-constrained environments.

Abstract

Large language models (LLMs) are increasingly deployed with task-specific adapters catering to multiple downstream applications. In such a scenario, the additional compute associated with these apparently insignificant number of adapter parameters (typically less than 1% of the base model) turns out to be disproportionately significant during inference time (upto 2.5x times that of the base model). In this paper, we propose a new zero-latency fused low-rank adapter (zFLoRA) that introduces zero or negligible latency overhead on top of the base model. Experimental results on LLMs of size 1B, 3B and 7B show that zFLoRA compares favorably against the popular supervised fine-tuning benchmarks including low-rank adapters (LoRA) as well as full fine-tuning (FFT). Experiments are conducted on 18 different tasks across three different categories namely commonsense reasoning, math reasoning and summary-dialogue. Latency measurements made on NPU (Samsung Galaxy S25+) as well as GPU (NVIDIA H100) platforms show that the proposed zFLoRA adapters introduce zero to negligible latency overhead.

zFLoRA: Zero-Latency Fused Low-Rank Adapters

TL;DR

The paper tackles the latency overhead of adapter-based PEFT methods in LLM inference by introducing zFLoRA, a zero-latency fused low-rank adapter that merges adapter computations with base model projections. It designs a fusion strategy that expands input from to once and merges back to only at the LM head, enabling cross-block fusion across MHA and FFN while avoiding per-layer input-expansion and output-merge costs. Empirically, zFLoRA matches the performance of LoRA and full fine-tuning (FFT) across 18 tasks on 1B, 3B, and 7B LLaMA models, while delivering negligible latency on both GPU (NVIDIA H100, via vLLM) and mobile NPU (Galaxy S25+), underscoring its practicality for on-device and edge deployments. The results indicate that zFLoRA provides a practical path to fast, task-specific fine-tuning with minimal compute overhead, enabling broader deployment of adaptable LLMs in latency-constrained environments.

Abstract

Large language models (LLMs) are increasingly deployed with task-specific adapters catering to multiple downstream applications. In such a scenario, the additional compute associated with these apparently insignificant number of adapter parameters (typically less than 1% of the base model) turns out to be disproportionately significant during inference time (upto 2.5x times that of the base model). In this paper, we propose a new zero-latency fused low-rank adapter (zFLoRA) that introduces zero or negligible latency overhead on top of the base model. Experimental results on LLMs of size 1B, 3B and 7B show that zFLoRA compares favorably against the popular supervised fine-tuning benchmarks including low-rank adapters (LoRA) as well as full fine-tuning (FFT). Experiments are conducted on 18 different tasks across three different categories namely commonsense reasoning, math reasoning and summary-dialogue. Latency measurements made on NPU (Samsung Galaxy S25+) as well as GPU (NVIDIA H100) platforms show that the proposed zFLoRA adapters introduce zero to negligible latency overhead.

Paper Structure

This paper contains 23 sections, 2 equations, 7 figures, 12 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Family of fused adapters
  4. Partially-fused LoRA
  5. Fused forward adapters
  6. Fused backward adapters
  7. Fused forward-backward adapters
  8. Zero-latency fused low-rank adapters
  9. Experiments and results
  10. Datasets
  11. Experimental setup
  12. Results on 1B and 3B models
  13. Scaling up and comparison experiments
  14. Latency measurements
  15. Latencies on H100 GPU
  16. ...and 8 more sections

Figures (7)

  • Figure 1: Inference latencies (first-token and per-token) of LoRA and zFLoRA for different input prompt lengths (512 to 2048) using vllm inference engine on NVIDIA H100 GPU at FP16 precision, expressed as a percentage of the base model (LLaMA 1B, 3B and 8B) latencies.
  • Figure 2: Block schematic of LoRA, and the basic building blocks of a fused adapter (F-Adapter and B-Adapter) for a single projection layer.
  • Figure 3: Single layer adapter latency simulations for base model layer, LoRA, pfLoRA and a fused layer.
  • Figure 4: Block schematic of zFLoRA architecture within a single transformer block or layer.
  • Figure 5: On-device prefill and decode latencies of LoRA and zFLoRA for varying prompt lengths (top row) and adapter ranks (bottom row), as compared to the base model (1B) on Samsung Galaxy S25+ mobile handset.
  • ...and 2 more figures