SineLoRA$Δ$: Sine-Activated Delta Compression

TL;DR

The paper tackles Delta Compression for resource-constrained model adaptation by addressing the expressivity gap of quantized Low-Rank Adapters. It introduces SineLoRAΔ, a rank-enhancing approach that applies a fixed-frequency sinusoidal activation after quantization, supported by a theoretical bound showing that stable rank under quantization is governed by the unquantized adapter and can be boosted with sine activations. The authors formalize the framework with a Main Theorem on stable rank, complement it with BD analysis to compare rate-distortion performance, and validate the method across large language models, vision-language tasks, and text-to-image generation, achieving memory reductions up to 66% with competitive accuracy. The work demonstrates practical gains for bandwidth- and memory-constrained deployments and proposes a principled evaluation protocol for PEFT compression using Bjøntegaard Delta analysis, while highlighting avenues for quantization-aware training and inference optimizations.

Abstract

Resource-constrained weight deployment is a task of immense practical importance. Recently, there has been interest in the specific task of \textit{Delta Compression}, where parties each hold a common base model and only communicate compressed weight updates. However, popular parameter efficient updates such as Low Rank Adaptation (LoRA) face inherent representation limitations - which are especially pronounced when combined with aggressive quantization. To overcome this, we build on recent work that improves LoRA representation capacity by using fixed-frequency sinusoidal functions to increase stable rank without adding additional parameters. We extend this to the quantized setting and present the first theoretical analysis showing how stable rank evolves under quantization. From this, we introduce SineLoRA$Δ$, a principled and effective method for delta compression that improves the expressivity of quantized low-rank adapters by applying a sinusoidal activation. We validate SineLoRA$Δ$ across a diverse variety of domains - including language modeling, vision-language tasks, and text-to-image generation - achieving up to 66% memory reduction with similar performance. We additionally provide a novel application of the canonical Bjøntegaard Delta metric to consistently compare adapter compression changes across the rate-distortion curve.

Figures (5)

• Figure 1: Effects of changing the $\omega$ term in a sine-activated low-rank matrix. Stable Rank saturates at sufficient $\omega$.
• Figure 2: A sine-activated low-rank matrix $\sin(\omega AB)$ increases the stable rank relative to a low-rank matrix $AB$. By varying quantization level $\sin(\omega(Q(A)Q(B))$ we can interpolate the effect on stable rank between these two values.
• Figure 3: Bjøntegaard Delta is calculated by taking (bitrate, accuracy) pairs for two codecs, and evaluating the (horizontal or vertical) integral between interpolated performance.
• Figure 4: Commonsense Reasoning performance (average) for SineLoRA$\Delta$ and LoRA with a frozen non-quantized LLAMA-3-8B base model. SineLoRA$\Delta$ exceeds the benchmark LoRA performance across all evaluated rank and quantization levels.
• Figure 5:

Theorems & Definitions (2)

• Theorem 1
• proof