Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Robustness and Regularization in Hierarchical Re-Basin

Benedikt Franke, Florian Heinrich, Markus Lange, Arne Raulf

TL;DR

The paper addresses the challenge of merging many trained networks by introducing a hierarchical Git Re-Basin scheme that combines $2^n$ models across $n$ stages using pairwise teleports and a $0.5$ interpolation. It demonstrates that this hierarchical approach outperforms the existing MergeMany method and can induce adversarial and perturbation robustness—scaling with the number of merged models—while also reducing weight norms and Lipschitz bounds, albeit at the cost of accuracy relative to the unmerged baseline. The work also highlights inconsistencies with prior reports of a zero-accuracy barrier and reproducibility, emphasizing the need for more rigorous evaluation. Overall, the proposed method offers a scalable, robustness-aware fusion mechanism and contributes to the understanding of permutation-invariant model merging and linear mode connectivity.

Abstract

This paper takes a closer look at Git Re-Basin, an interesting new approach to merge trained models. We propose a hierarchical model merging scheme that significantly outperforms the standard MergeMany algorithm. With our new algorithm, we find that Re-Basin induces adversarial and perturbation robustness into the merged models, with the effect becoming stronger the more models participate in the hierarchical merging scheme. However, in our experiments Re-Basin induces a much bigger performance drop than reported by the original authors.

Robustness and Regularization in Hierarchical Re-Basin

TL;DR

The paper addresses the challenge of merging many trained networks by introducing a hierarchical Git Re-Basin scheme that combines models across stages using pairwise teleports and a interpolation. It demonstrates that this hierarchical approach outperforms the existing MergeMany method and can induce adversarial and perturbation robustness—scaling with the number of merged models—while also reducing weight norms and Lipschitz bounds, albeit at the cost of accuracy relative to the unmerged baseline. The work also highlights inconsistencies with prior reports of a zero-accuracy barrier and reproducibility, emphasizing the need for more rigorous evaluation. Overall, the proposed method offers a scalable, robustness-aware fusion mechanism and contributes to the understanding of permutation-invariant model merging and linear mode connectivity.

Abstract

This paper takes a closer look at Git Re-Basin, an interesting new approach to merge trained models. We propose a hierarchical model merging scheme that significantly outperforms the standard MergeMany algorithm. With our new algorithm, we find that Re-Basin induces adversarial and perturbation robustness into the merged models, with the effect becoming stronger the more models participate in the hierarchical merging scheme. However, in our experiments Re-Basin induces a much bigger performance drop than reported by the original authors.

Paper Structure

This paper contains 7 sections, 5 figures.

Table of Contents

  1. Introduction
  2. Related Work
  3. Our Method
  4. Experiments
  5. Robustness and Relationship with Regularization
  6. Effects on Model Norm and Lipschitz Constant
  7. Discussion

Figures (5)

  • Figure 1: Our proposed hierarchical Merging Scheme, exemplified for merging eight models.
  • Figure 2: Distribution of test set accuracies on CIFAR-10 by merging algorithm on 1600 trained input models. "No Merging" denotes the median accuracy of the trained input models and is displayed only as a line as the unmerged models have very little variance to improve the clarity of the figure.
  • Figure 3: Mean Accuracy of different Re-Basin stages over attack strength $\epsilon$. As a comparison, models trained with L1 and L2 regularization are also shown. 95% CI is shaded around the curves but is too small to be visible.
  • Figure 4: Impact of Different Re-Basin Stages on Weight Norm and Lipschitz Upper Bound
  • Figure :