LoRA-Drop: Temporal LoRA Decoding for Efficient LLM Inference

TL;DR

LoRA-Drop targets the latency bottleneck in autoregressive LLM inference by temporally scheduling a fixed subset of transformer layers to run in a lightweight LoRA mode, with periodic full-refresh steps to prevent drift. By exploiting temporal redundancy in hidden states, it reduces per-token compute and KV-cache updates without routing networks, and is designed as a plug-and-play enhancement for pretrained models. Across four open-weight LLMs, it delivers up to 2.6x decoding speedups and 40–55% KV-cache reductions with less than 0.5 percentage-point accuracy loss on reasoning, code, and multilingual benchmarks, establishing a practical Pareto frontier for adaptive-capacity inference. The approach relies on a model-specific drop-layer list derived from empirical redundancy measurements and requires only modest continual fine-tuning of LoRA parameters, enabling straightforward deployment on constrained hardware.

Abstract

Autoregressive large language models (LLMs) are bottlenecked by sequential decoding, where each new token typically requires executing all transformer layers. Existing dynamic-depth and layer-skipping methods reduce this cost, but often rely on auxiliary routing mechanisms or incur accuracy degradation when bypassed layers are left uncompensated. We present \textbf{LoRA-Drop}, a plug-and-play inference framework that accelerates decoding by applying a \emph{temporal compute schedule} to a fixed subset of intermediate layers: on most decoding steps, selected layers reuse the previous-token hidden state and apply a low-rank LoRA correction, while periodic \emph{refresh} steps execute the full model to prevent drift. LoRA-Drop requires no routing network, is compatible with standard KV caching, and can reduce KV-cache footprint by skipping KV updates in droppable layers during LoRA steps and refreshing periodically. Across \textbf{LLaMA2-7B}, \textbf{LLaMA3-8B}, \textbf{Qwen2.5-7B}, and \textbf{Qwen2.5-14B}, LoRA-Drop achieves up to \textbf{2.6$\times$ faster decoding} and \textbf{45--55\% KV-cache reduction} while staying within \textbf{0.5 percentage points (pp)} of baseline accuracy. Evaluations on reasoning (GSM8K, MATH, BBH), code generation (HumanEval, MBPP), and long-context/multilingual benchmarks (LongBench, XNLI, XCOPA) identify a consistent \emph{safe zone} of scheduling configurations that preserves quality while delivering substantial efficiency gains, providing a simple path toward adaptive-capacity inference in LLMs. Codes are available at https://github.com/hosseinbv/LoRA-Drop.git.

Figures (4)

• Figure 1: (Left) Adjacent-token redundancy measured as the average cosine similarity between the hidden state of token $t$ and those of the next tokens $t+\Delta$ (with $\Delta \in \{1,...,10\}$), averaged over all positions $t$ across 1024 batches of diverse data. All evaluated models show high similarity (approximately $0.6$--$0.85$), indicating that hidden states change little between consecutive tokens. (Right) The similarity horizon, defined as the largest token distance $\Delta$ for which the cosine similarity between token $t$ and $t+\Delta$ remains at least $0.50$. Horizons between $3$ and $6$ tokens demonstrate that several future tokens share highly similar hidden states.
• Figure 2: The workflow of LoRA-Drop. Each transformer layer is augmented with a low-rank adaptation module (matrices $A_i$ and $B_i$). During low-complexity steps (eg. $t+1$ and $t+2$), only the lightweight LoRA modules are activated, bypassing the pre-specified layers to reduce inference computation. At periodic or complexity-triggered steps, the full layers are reactivated to refine representations.
• Figure 3: KV-cache savings vs. drop-ratio and temporal window $k$ (2$\times$2 grid). Each curve shows the estimated percentage reduction in KV memory relative to the full-model baseline when generating $N=32\text{k}$ tokens, for $k\in\{1,2,3,4,5\}$. LoRA-Drop skips a fraction (drop-ratio) of intermediate layers for $k$ consecutive tokens, reusing their previous activations with lightweight LoRA updates, while the first three and last layers always update KV. Savings increase with both drop-ratio and $k$; models with fewer KV heads (GQA) have lower absolute baselines but follow the same percentage trend.
• Figure 4: Similarity decay across layers for fixed token distances $\Delta$. Each curve corresponds to $\mathrm{sim}(\ell,\Delta)$ for a particular $\Delta \in \{1,2,3,5,10\}$. Across all models, adjacent-token similarity remains extremely high in early layers (0.8--0.95), decreases gradually in middle layers, and sometimes rises again near top layers. Similarity drops sharply for larger $\Delta$ values, but non-negligible redundancy persists up to $\Delta = 3$ in several architectures.