Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

BD-Merging: Bias-Aware Dynamic Model Merging with Evidence-Guided Contrastive Learning

Yuhan Xie, Chen Lyu

TL;DR

BD-Merging is presented, a bias-aware unsupervised model merging framework that explicitly models uncertainty to achieve adaptive reliability under distribution shift and introduces a joint evidential head that learns uncertainty over a unified label space, capturing cross-task semantic dependencies in MM.

Abstract

Model Merging (MM) has emerged as a scalable paradigm for multi-task learning (MTL), enabling multiple task-specific models to be integrated without revisiting the original training data. Despite recent progress, the reliability of MM under test-time distribution shift remains insufficiently understood. Most existing MM methods typically assume that test data are clean and distributionally aligned with both the training and auxiliary sources. However, this assumption rarely holds in practice, often resulting in biased predictions with degraded generalization. To address this issue, we present BD-Merging, a bias-aware unsupervised model merging framework that explicitly models uncertainty to achieve adaptive reliability under distribution shift. First, BD-Merging introduces a joint evidential head that learns uncertainty over a unified label space, capturing cross-task semantic dependencies in MM. Second, building upon this evidential foundation, we propose an Adjacency Discrepancy Score (ADS) that quantifies evidential alignment among neighboring samples. Third, guided by ADS, a discrepancy-aware contrastive learning mechanism refines the merged representation by aligning consistent samples and separating conflicting ones. Combined with general unsupervised learning, this process trains a debiased router that adaptively allocates task-specific or layer-specific weights on a per-sample basis, effectively mitigating the adverse effects of distribution shift. Extensive experiments across diverse tasks demonstrate that BD-Merging achieves superior effectiveness and robustness compared to state-of-the-art MM baselines.

BD-Merging: Bias-Aware Dynamic Model Merging with Evidence-Guided Contrastive Learning

TL;DR

BD-Merging is presented, a bias-aware unsupervised model merging framework that explicitly models uncertainty to achieve adaptive reliability under distribution shift and introduces a joint evidential head that learns uncertainty over a unified label space, capturing cross-task semantic dependencies in MM.

Abstract

Model Merging (MM) has emerged as a scalable paradigm for multi-task learning (MTL), enabling multiple task-specific models to be integrated without revisiting the original training data. Despite recent progress, the reliability of MM under test-time distribution shift remains insufficiently understood. Most existing MM methods typically assume that test data are clean and distributionally aligned with both the training and auxiliary sources. However, this assumption rarely holds in practice, often resulting in biased predictions with degraded generalization. To address this issue, we present BD-Merging, a bias-aware unsupervised model merging framework that explicitly models uncertainty to achieve adaptive reliability under distribution shift. First, BD-Merging introduces a joint evidential head that learns uncertainty over a unified label space, capturing cross-task semantic dependencies in MM. Second, building upon this evidential foundation, we propose an Adjacency Discrepancy Score (ADS) that quantifies evidential alignment among neighboring samples. Third, guided by ADS, a discrepancy-aware contrastive learning mechanism refines the merged representation by aligning consistent samples and separating conflicting ones. Combined with general unsupervised learning, this process trains a debiased router that adaptively allocates task-specific or layer-specific weights on a per-sample basis, effectively mitigating the adverse effects of distribution shift. Extensive experiments across diverse tasks demonstrate that BD-Merging achieves superior effectiveness and robustness compared to state-of-the-art MM baselines.
Paper Structure (29 sections, 17 equations, 6 figures, 3 tables, 1 algorithm)

This paper contains 29 sections, 17 equations, 6 figures, 3 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Model Merging
  4. Evidential Deep Learning
  5. Preliminaries
  6. Problem Setup
  7. Methodology
  8. Dirichlet-Based Evidential Modeling
  9. Joint Evidential Head
  10. Adjacency Set Construction
  11. Prediction Sharpness.
  12. Semantic Divergence.
  13. Opinion Conflicts.
  14. Discrepancy-Aware Contrastive Merging
  15. Debiased Router.
  16. ...and 14 more sections

Figures (6)

  • Figure 1: Averaged normalized accuracy under different distribution shift settings across multiple benchmarks.
  • Figure 2: The process of BD-Merging. BD-Merging includes three modules: (1) Modeling uncertainty using a Dirichlet-based evidential head; (2) Constructing an adjacency set and calculating the ADS to measure evidential alignment; and (3) Performing discrepancy-aware contrastive MM with a debiased router, guided by ADS. Stage (a)-(d) demonstrates the evolution of samples in the feature space through various stages.
  • Figure 3: Visualization of corrupted image instances.
  • Figure 4: Evaluation of BD-Merging under test-time bias across various models.
  • Figure 5: Performance vs. time cost of MM methods.
  • ...and 1 more figures

Theorems & Definitions (2)

  • Definition 1: Dirichlet Distribution
  • Definition 2: Dirichlet-Based Evidential Deep Learning