RanDeS: Randomized Delta Superposition for Multi-Model Compression

TL;DR

This paper tackles the memory bottleneck of serving many fine-tuned models by treating model merging as a compress-and-retrieve problem and identifying task interference as the accumulation of conflicting deltas. It introduces Randomized Delta Superposition (RanDeS), which decorrelates task deltas via random orthogonal transformations and retrieves per-task models through inverse transforms, all without retraining. Two practical layer-wise implementations—layer shuffling (RanDeS-S) and layer column-wise random sign flips (RanDeS-RSF or with both as RanDeS-SRSF)—yield substantial accuracy gains over prior merging methods while keeping memory overhead low, thanks to seed-defined transforms. Experiments across CLIP-ViT, Flan-T5 with LoRA, and GPT-2 demonstrate strong cross-task performance, near-fine-tuned accuracy in several benchmarks, and scalable memory usage up to 20 models, making RanDeS practical for dynamic multi-model serving. The work also shows effective applicability to PEFT model compression and real-time model hot-swapping, highlighting its potential to significantly reduce deployment costs in multi-task AI systems.

Abstract

From a multi-model compression perspective, model merging enables memory-efficient serving of multiple models fine-tuned from the same base, but suffers from degraded performance due to interference among their task-specific parameter adjustments (i.e., deltas). In this paper, we reformulate model merging as a compress-and-retrieve scheme, revealing that the task interference arises from the summation of irrelevant deltas during model retrieval. To address this issue, we use random orthogonal transformations to decorrelate these vectors into self-cancellation. We show that this approach drastically reduces interference, improving performance across both vision and language tasks. Since these transformations are fully defined by random seeds, adding new models requires no extra memory. Further, their data- and model-agnostic nature enables easy addition or removal of models with minimal compute overhead, supporting efficient and flexible multi-model serving.

Figures (7)

• Figure 1: Illustration of RanDeS with three deltas $\{\bm{\Delta}_i\}_{i=1}^3$.
• Figure 2: Cosine similarity distributions between delta layers within and across models for CLIP-ViT-B/32, Flan-T5, and GPT-2.
• Figure 3: Average pairwise cosine similarity of three out of eight CLIP-ViT-B/32 task vectors during model retrieval for SUN397 across three repetitions. Both RanDeS-S and RanDeS-RSF increase mutual orthogonality, with an additive effect when combined.
• Figure 4: The impact of $\lambda$ on average accuracy over eight image classification tasks.
• Figure 5: (a) Impact of context matrix design to the average accuracy. RBD stands for random binary diagonal; RD stands for random diagonal. (b) Impact of target layer selection to the average accuracy. ALL stands for choosing all layers; MLP means only MLP layers are selected; and ATTN stands for attention layers.