Long Exposure: Accelerating Parameter-Efficient Fine-Tuning for LLMs under Shadowy Sparsity

TL;DR

Long Exposure identifies shadowy sparsity as a pervasive, overlapping sparse pattern in LLM fine-tuning and targets acceleration of parameter-efficient fine-tuning (PEFT). It introduces three components—Shadowy-sparsity Exposer, Sequence-oriented Predictor, and Dynamic-aware Operator—to capture, predict, and exploit these patterns for per-layer speedups in multi-head attention and MLP blocks. The system achieves substantial end-to-end speedups (up to about 2.5x) and memory savings while preserving accuracy across multiple PEFT methods and model scales, demonstrating practical viability for faster PEFT of LLMs. By bridging sparsity insights from inference to training, Long Exposure offers a scalable, hardware-conscious approach to reduce wall-clock time for PEFT, enabling more cost-effective deployment and iteration in downstream tasks.

Abstract

The adaptation of pre-trained large language models (LLMs) to diverse downstream tasks via fine-tuning is critical for numerous applications. However, the inefficiency of parameter-efficient fine-tuning (PEFT) techniques presents significant challenges in terms of time investments and operational costs. In this paper, we first introduce a nuanced form of sparsity, termed Shadowy Sparsity, which is distinctive in fine-tuning and has not been adequately addressed for acceleration. Under Shadowy Sparsity, we propose Long Exposure, an efficient system to accelerate PEFT for LLMs. Long Exposure comprises three key components: Shadowy-sparsity Exposer employs a prolonged sensing range to capture more sparsity details under shadowy sparsity; Sequence-oriented Predictor provides efficient yet accurate predictions to handle large sequence inputs and constantly-evolving parameters; and Dynamic-aware Operator facilitates more structured computational patterns and coalesced memory accesses, addressing dynamic sparse operations. Extensive evaluations show that Long Exposure outperforms state-of-the-arts with up to a $2.49\times$ speedup in end-to-end fine-tuning, offering promising advancements in accelerating PEFT for LLMs.

Figures (14)

• Figure 1: Shadowy sparsity in LLM fine-tuning. Transformer-based models are generally composed of two primary components: multi-head attention and MLP block. In fine-tuning, the sparse patterns of various tokens within the input sequence exhibit a logical AND relationship, which restricts the sparsity degree and leads to computational waste in both components.
• Figure 2: An illustrated example of LoRA for showcasing the computational flow (forward and backward) in PEFT.
• Figure 3: Long Exposure overview
• Figure 4: Visualization of attention scores in multi-head attention and activations in MLP block. Due to varying input lengths, these metrics manifest as vectors during inference and as matrices during fine-tuning. Brighter colors denote high values.
• Figure 5: The process of prediction (batch size = 1 for simplicity). Each predictor comprises a set of trainable parameters (green), which accepts tensors (blue) as input and produces approximations of attention scores or activations. These outcomes are then processed to filter out less significant values (light orange), and a reduction is performed if needed, to generate the final sparse pattern (orange) for the token sequence.