Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Modular Delta Merging with Orthogonal Constraints: A Scalable Framework for Continual and Reversible Model Composition

Haris Khan, Sadia Asif, Shumaila Asif

TL;DR

Modular Delta Merging with Orthogonal Constraints (MDM-OC), a novel framework that enables scalable, interference-free, and reversible composition of fine-tuned models, offers a principled solution for modular and compliant AI system design.

Abstract

In real-world machine learning deployments, models must be continually updated, composed, and when required, selectively undone. However, existing approaches to model merging and continual learning often suffer from task interference, catastrophic forgetting, or lack of reversibility. We propose Modular Delta Merging with Orthogonal Constraints (MDM-OC), a novel framework that enables scalable, interference-free, and reversible composition of fine-tuned models. Each task-specific model is encoded as a delta from a shared base and projected into an orthogonal subspace to eliminate conflict. These projected deltas are then merged via gradient-based optimization to form a unified model that retains performance across tasks. Our approach supports continual integration of new models, structured unmerging for compliance such as GDPR requirements, and model stability via elastic weight consolidation and synthetic replay. Extensive experiments on vision and natural language processing benchmarks demonstrate that MDM-OC outperforms prior baselines in accuracy, backward transfer, and unmerge fidelity, while remaining memory-efficient and computationally tractable. This framework offers a principled solution for modular and compliant AI system design.

Modular Delta Merging with Orthogonal Constraints: A Scalable Framework for Continual and Reversible Model Composition

TL;DR

Modular Delta Merging with Orthogonal Constraints (MDM-OC), a novel framework that enables scalable, interference-free, and reversible composition of fine-tuned models, offers a principled solution for modular and compliant AI system design.

Abstract

In real-world machine learning deployments, models must be continually updated, composed, and when required, selectively undone. However, existing approaches to model merging and continual learning often suffer from task interference, catastrophic forgetting, or lack of reversibility. We propose Modular Delta Merging with Orthogonal Constraints (MDM-OC), a novel framework that enables scalable, interference-free, and reversible composition of fine-tuned models. Each task-specific model is encoded as a delta from a shared base and projected into an orthogonal subspace to eliminate conflict. These projected deltas are then merged via gradient-based optimization to form a unified model that retains performance across tasks. Our approach supports continual integration of new models, structured unmerging for compliance such as GDPR requirements, and model stability via elastic weight consolidation and synthetic replay. Extensive experiments on vision and natural language processing benchmarks demonstrate that MDM-OC outperforms prior baselines in accuracy, backward transfer, and unmerge fidelity, while remaining memory-efficient and computationally tractable. This framework offers a principled solution for modular and compliant AI system design.

Paper Structure

This paper contains 44 sections, 3 theorems, 24 equations, 6 figures, 2 tables.

Table of Contents

  1. INTRODUCTION
  2. RELATED WORK
  3. Continual Learning and Catastrophic Forgetting
  4. Model Merging and Parameter Composition
  5. Orthogonal Projections and Interference Reduction
  6. Knowledge Editing and Reversible Updates
  7. PROBLEM FORMULATION
  8. Mathematical Framework
  9. Delta Representation
  10. Orthogonality Constraints
  11. Continual Integration and Unmerging
  12. Efficiency and Scalability
  13. Performance Bound
  14. PROPOSED METHOD
  15. Delta Extraction
  16. ...and 29 more sections

Key Result

Theorem V.1

Let $\{\Delta \theta_1, \ldots, \Delta \theta_N\}$ denote task-specific deltas and $\{\Delta \theta_1^{\perp}, \ldots, \Delta \theta_N^{\perp}\}$ their orthogonalized forms from the Gram--Schmidt process: Then, $\langle \Delta \theta_i^{\perp}, \Delta \theta_j^{\perp} \rangle = 0$ for all $i \neq j$.

Figures (6)

  • Figure 1: Overview of MDM-OC: task-specific deltas are orthogonalized and merged via optimized coefficients, supporting continual integration and reversible unmerging.
  • Figure 2: PCA projection of task deltas before (red) and after (green) orthogonalization, showing reduced overlap.
  • Figure 3: CMA-ES optimization trajectory over $\alpha$ coefficients toward balanced multi-task performance.
  • Figure 4: Unmerging a single task while maintaining others' accuracy.
  • Figure 5: Comparison across CIFAR-100, ImageNet-100, and multi-domain text tasks. MDM-OC achieves highest average accuracy, highlighting stability and plasticity through orthogonal task integration.
  • ...and 1 more figures

Theorems & Definitions (3)

  • Theorem V.1: Orthogonality Preservation
  • Corollary V.2: Span Preservation
  • Theorem V.3: Bounded Numerical Interference