Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

MINGLE: Mixture of Null-Space Gated Low-Rank Experts for Test-Time Continual Model Merging

Zihuan Qiu, Yi Xu, Chiyuan He, Fanman Meng, Linfeng Xu, Qingbo Wu, Hongliang Li

TL;DR

This paper introduces Test-Time Continual Model Merging (TTCMM) to fuse independently fine-tuned models during inference without training data access. It presents MINGLE, a mixture of null-space gated low-rank experts that uses seed unlabeled samples for test-time adaptation, with an adaptive relaxation strategy to balance stability and plasticity, and a null-space constraint to suppress interference from previously learned tasks. Theoretical analysis motivates dynamic routing over static averaging, and extensive experiments on vision and NLP benchmarks show that MINGLE achieves 7–9% higher accuracy on average with near-zero forgetting across varied task orders, while maintaining efficient parameter usage. This work demonstrates that TTCMM with adaptive null-space gating offers robust generalization under distribution shifts and privacy-preserving model merging, with practical applicability across diverse domains.

Abstract

Continual model merging integrates independently fine-tuned models sequentially without access to the original training data, offering a scalable and efficient solution for continual learning. However, existing methods face two critical challenges: parameter interference among tasks, which leads to catastrophic forgetting, and limited adaptability to evolving test distributions. To address these issues, we introduce the task of Test-Time Continual Model Merging (TTCMM), which leverages a small set of unlabeled test samples during inference to alleviate parameter conflicts and handle distribution shifts. We propose MINGLE, a novel framework for TTCMM. MINGLE employs a mixture-of-experts architecture with parameter-efficient, low-rank experts, which enhances adaptability to evolving test distributions while dynamically merging models to mitigate conflicts. To further reduce forgetting, we propose Null-Space Constrained Gating, which restricts gating updates to subspaces orthogonal to prior task representations, thereby suppressing activations on old tasks and preserving past knowledge. We further introduce an Adaptive Relaxation Strategy that adjusts constraint strength dynamically based on interference signals observed during test-time adaptation, striking a balance between stability and adaptability. Extensive experiments on standard continual merging benchmarks demonstrate that MINGLE achieves robust generalization, significantly reduces forgetting, and consistently surpasses previous state-of-the-art methods by 7-9% on average across diverse task orders. Our code is available at: https://github.com/zihuanqiu/MINGLE

MINGLE: Mixture of Null-Space Gated Low-Rank Experts for Test-Time Continual Model Merging

TL;DR

This paper introduces Test-Time Continual Model Merging (TTCMM) to fuse independently fine-tuned models during inference without training data access. It presents MINGLE, a mixture of null-space gated low-rank experts that uses seed unlabeled samples for test-time adaptation, with an adaptive relaxation strategy to balance stability and plasticity, and a null-space constraint to suppress interference from previously learned tasks. Theoretical analysis motivates dynamic routing over static averaging, and extensive experiments on vision and NLP benchmarks show that MINGLE achieves 7–9% higher accuracy on average with near-zero forgetting across varied task orders, while maintaining efficient parameter usage. This work demonstrates that TTCMM with adaptive null-space gating offers robust generalization under distribution shifts and privacy-preserving model merging, with practical applicability across diverse domains.

Abstract

Continual model merging integrates independently fine-tuned models sequentially without access to the original training data, offering a scalable and efficient solution for continual learning. However, existing methods face two critical challenges: parameter interference among tasks, which leads to catastrophic forgetting, and limited adaptability to evolving test distributions. To address these issues, we introduce the task of Test-Time Continual Model Merging (TTCMM), which leverages a small set of unlabeled test samples during inference to alleviate parameter conflicts and handle distribution shifts. We propose MINGLE, a novel framework for TTCMM. MINGLE employs a mixture-of-experts architecture with parameter-efficient, low-rank experts, which enhances adaptability to evolving test distributions while dynamically merging models to mitigate conflicts. To further reduce forgetting, we propose Null-Space Constrained Gating, which restricts gating updates to subspaces orthogonal to prior task representations, thereby suppressing activations on old tasks and preserving past knowledge. We further introduce an Adaptive Relaxation Strategy that adjusts constraint strength dynamically based on interference signals observed during test-time adaptation, striking a balance between stability and adaptability. Extensive experiments on standard continual merging benchmarks demonstrate that MINGLE achieves robust generalization, significantly reduces forgetting, and consistently surpasses previous state-of-the-art methods by 7-9% on average across diverse task orders. Our code is available at: https://github.com/zihuanqiu/MINGLE
Paper Structure (37 sections, 2 theorems, 30 equations, 9 figures, 16 tables, 1 algorithm)

This paper contains 37 sections, 2 theorems, 30 equations, 9 figures, 16 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Mingle: Mixture of Null-Space Gated Low-Rank Experts
  4. Preliminaries
  5. Problem Setting.
  6. Motivation and Theoretical Analysis
  7. Low-Rank Expert Mixture for Continual Model Merging
  8. Adaptive Null-Space Constrained Gating for Interference Mitigation
  9. Experiments
  10. Experimental Setup
  11. Main Results
  12. Ablation Results and Analysis
  13. Conclusions
  14. Acknowledgments
  15. Theoretical Risk Comparison: Dynamic MoE vs. Static Averaging
  16. ...and 22 more sections

Key Result

Theorem 1

Let $\{(D_t,f_t)\}_{t=1}^T$ be $T$ independent tasks with priors $P(t)$ and per-task risks $R_t(i)$. For any static mixture $h_{\mathrm{static}}(x)=\sum_{i=1}^T\alpha_i\,f_i(x)$ and any hard-routed MoE $h_{\mathrm{MoE}}(x)=f_{i^\star(x)}(x)$ with task-specific routing errors $\varepsilon_t$: where $R_{\mathrm{ideal}}=\sum_tP(t)R_t(t)$ and $R_{\text{wrong},t}=\frac{1}{T-1}\sum_{i\neq t}R_t(i)$. Mo

Figures (9)

  • Figure 1: After 8-task continual merging: accuracy on first four tasks and overall BWT.
  • Figure 2: Comparison of three continual learning paradigms. (a) Conventional Continual Learning trains models sequentially with data arriving in stages, without access to previous task data. (b) Continual Model Merging continually fuses independently trained models, without access to any training data. (c) Test-Time Continual Model Merging improves merging by leveraging a few unlabeled test samples from the current task.
  • Figure 3: Gate activations across eight tasks under varying $\gamma$. Each subplot shows one gate; curves and shaded areas indicate mean and std across layers. Gray bars mark the gate’s training task. Lower $\gamma$ leads to stronger suppression on prior tasks.
  • Figure 4: Sensitivity analysis of the null-space constrained gating w.r.t. hyper-parameters $\beta$, $\gamma$, and $k$.
  • Figure 5: Accuracy matrices of Mingle (ViT-B/32, ViT-B/16, and ViT-L/14) under different task settings.
  • ...and 4 more figures

Theorems & Definitions (3)

  • Theorem 1: Dynamic MoE versus Static Averaging
  • Theorem A.1: Dynamic MoE versus Static Averaging
  • proof