Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Per-Axis Weight Deltas for Frequent Model Updates

Stefan Kuyumdzhiev, Radostin Cholakov

TL;DR

The paper addresses the challenge of deploying many task-specific fine-tuned LLM variants by compressing their deltas relative to a shared base with a 1-bit sign mask and per-axis scales: $\widehat{\mathbf{W}}^{(i)} = \mathbf{v}^{(i)} \odot \mathbf{B}^{(i)} + \mathbf{W_b}^{(i)}$ where $\mathbf{B}^{(i)} = \mathrm{sign}(\Delta \mathbf{W}^{(i)})$ and $\Delta \mathbf{W}^{(i)} = \mathbf{W_f}^{(i)} - \mathbf{W_b}^{(i)}$. The method includes calibration-driven per-layer scale learning, axis selection (row/column), and end-to-end fine-tuning of the scales, enabling accurate reconstruction with small artifacts. Empirically, it yields consistent improvements over both the uncompressed baseline and a scalar delta variant, while delivering substantial storage reductions (≈5×) and faster load times (≈0.8 s vs 2.1 s), enabling fast hot-swapping of many variants. This approach offers practical benefits for multi-tenant and continual adaptation scenarios by reducing cold-start latency without sacrificing inference efficiency, and points to future directions in more granular scaling and sign-structure learning.

Abstract

Serving many task-specialized LLM variants is often limited by the large size of fine-tuned checkpoints and the resulting cold-start latency. Since fine-tuned weights differ from their base model by relatively small structured residuals, a natural approach is to represent them as compressed deltas. We propose a simple 1-bit delta scheme that stores only the sign of the weight difference together with lightweight per-axis (row/column) FP16 scaling factors, learned from a small calibration set. This design preserves the compactness of 1-bit deltas while more accurately capturing variation across weight dimensions, leading to improved reconstruction quality over scalar alternatives. From a systems perspective, a streamlined loader that transfers packed deltas in a single operation per module reduces cold-start latency and storage overhead, with artifacts several times smaller than a full FP16 checkpoint. The method is drop-in, requires minimal calibration data, and maintains inference efficiency by avoiding dense reconstruction. Our experimental setup and source code are available at https://github.com/kuiumdjiev/Per-Axis-Weight-Deltas-for-Frequent-Model-Updates.

Per-Axis Weight Deltas for Frequent Model Updates

TL;DR

The paper addresses the challenge of deploying many task-specific fine-tuned LLM variants by compressing their deltas relative to a shared base with a 1-bit sign mask and per-axis scales: where and . The method includes calibration-driven per-layer scale learning, axis selection (row/column), and end-to-end fine-tuning of the scales, enabling accurate reconstruction with small artifacts. Empirically, it yields consistent improvements over both the uncompressed baseline and a scalar delta variant, while delivering substantial storage reductions (≈5×) and faster load times (≈0.8 s vs 2.1 s), enabling fast hot-swapping of many variants. This approach offers practical benefits for multi-tenant and continual adaptation scenarios by reducing cold-start latency without sacrificing inference efficiency, and points to future directions in more granular scaling and sign-structure learning.

Abstract

Serving many task-specialized LLM variants is often limited by the large size of fine-tuned checkpoints and the resulting cold-start latency. Since fine-tuned weights differ from their base model by relatively small structured residuals, a natural approach is to represent them as compressed deltas. We propose a simple 1-bit delta scheme that stores only the sign of the weight difference together with lightweight per-axis (row/column) FP16 scaling factors, learned from a small calibration set. This design preserves the compactness of 1-bit deltas while more accurately capturing variation across weight dimensions, leading to improved reconstruction quality over scalar alternatives. From a systems perspective, a streamlined loader that transfers packed deltas in a single operation per module reduces cold-start latency and storage overhead, with artifacts several times smaller than a full FP16 checkpoint. The method is drop-in, requires minimal calibration data, and maintains inference efficiency by avoiding dense reconstruction. Our experimental setup and source code are available at https://github.com/kuiumdjiev/Per-Axis-Weight-Deltas-for-Frequent-Model-Updates.
Paper Structure (15 sections, 2 equations, 2 figures, 2 tables, 7 algorithms)

This paper contains 15 sections, 2 equations, 2 figures, 2 tables, 7 algorithms.

Table of Contents

  1. Introduction
  2. Method
  3. Prior evidence against weight reconstruction.
  4. Calibration cache, training, and stacking.
  5. Implementation remarks.
  6. Experiments
  7. Setup
  8. Models and baselines.
  9. Main results
  10. Storage and load-time.
  11. Limitations
  12. Conclusion
  13. Acknowledgements
  14. Additional analysis of delta-quantization axis
  15. Counts by sub-type.

Figures (2)

  • Figure 1: Approximating the fine-tuned weights $\mathbf{W_f}$ by $\mathbf{v} \odot \mathbf{B} + \mathbf{W_b}$: a compact 1-bit sign residual, where $\mathbf{v}$ is a vector, $\mathbf{B} \in \{-1,+1\}$ is the binary sign matrix, and $\mathbf{W_b}$ is the base weight matrix.
  • Figure 2: counts of row vs. column delta-quantization per sub_type (row in blue, column in red).