Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

GLUE: Gradient-free Learning to Unify Experts

Jong-Ik Park, Shreyas Chaudhari, Srinivasa Pranav, Carlee Joe-Wong, José M. F. Moura

TL;DR

GLUE addresses the problem of constructing a strong target-domain prior when multiple pretrained experts exist by learning a convex combination of their parameters. It uses a gradient-free two-point SPSA update that operates entirely in the $K$-dimensional mixture space, requiring only forward passes and avoiding backpropagation through the blended model. The approach provides theoretical cost and variance insights and demonstrates empirical gains across three datasets and three architectures, closely matching full-gradient mixing while significantly reducing per-iteration computation. This yields a lightweight, deployable strategy for domain expansion and transfer learning in heterogeneous environments.

Abstract

In many deployed systems (multilingual ASR, cross-hospital imaging, region-specific perception), multiple pretrained specialist models coexist. Yet, new target domains often require domain expansion: a generalized model that performs well beyond any single specialist's domain. Given such a new target domain, prior works seek a single strong initialization prior for the model parameters by first blending expert models to initialize a target model. However, heuristic blending -- using coefficients based on data size or proxy metrics -- often yields lower target-domain test accuracy, and learning the coefficients on the target loss typically requires computationally-expensive full backpropagation through the network. We propose GLUE, Gradient-free Learning To Unify Experts, which initializes the target model as a convex combination of fixed experts, learning the mixture coefficients of this combination via a gradient-free two-point (SPSA) update that requires only two forward passes per step. Across experiments on three datasets and three network architectures, GLUE produces a single prior that can be fine-tuned effectively to outperform baselines. GLUE improves test accuracy by up to 8.5% over data-size weighting and by up to 9.1% over proxy-metric selection. GLUE either outperforms backpropagation-based full-gradient mixing or matches its performance within 1.4%.

GLUE: Gradient-free Learning to Unify Experts

TL;DR

GLUE addresses the problem of constructing a strong target-domain prior when multiple pretrained experts exist by learning a convex combination of their parameters. It uses a gradient-free two-point SPSA update that operates entirely in the -dimensional mixture space, requiring only forward passes and avoiding backpropagation through the blended model. The approach provides theoretical cost and variance insights and demonstrates empirical gains across three datasets and three architectures, closely matching full-gradient mixing while significantly reducing per-iteration computation. This yields a lightweight, deployable strategy for domain expansion and transfer learning in heterogeneous environments.

Abstract

In many deployed systems (multilingual ASR, cross-hospital imaging, region-specific perception), multiple pretrained specialist models coexist. Yet, new target domains often require domain expansion: a generalized model that performs well beyond any single specialist's domain. Given such a new target domain, prior works seek a single strong initialization prior for the model parameters by first blending expert models to initialize a target model. However, heuristic blending -- using coefficients based on data size or proxy metrics -- often yields lower target-domain test accuracy, and learning the coefficients on the target loss typically requires computationally-expensive full backpropagation through the network. We propose GLUE, Gradient-free Learning To Unify Experts, which initializes the target model as a convex combination of fixed experts, learning the mixture coefficients of this combination via a gradient-free two-point (SPSA) update that requires only two forward passes per step. Across experiments on three datasets and three network architectures, GLUE produces a single prior that can be fine-tuned effectively to outperform baselines. GLUE improves test accuracy by up to 8.5% over data-size weighting and by up to 9.1% over proxy-metric selection. GLUE either outperforms backpropagation-based full-gradient mixing or matches its performance within 1.4%.
Paper Structure (12 sections, 1 theorem, 17 equations, 1 figure, 1 table)

This paper contains 12 sections, 1 theorem, 17 equations, 1 figure, 1 table.

Table of Contents

  1. Introduction
  2. Related Work
  3. Methodology
  4. Objective
  5. Two-point SPSA updates
  6. From mixture to prior
  7. Theoretical Analysis
  8. Two-point (SPSA) vs. full-gradient cost for updating
  9. Variance of two-point estimates
  10. Experimental Evaluations
  11. Discussion
  12. Conclusion

Key Result

Proposition 1

The variance of the two-point (SPSA) gradient estimator $\overline{\mathbf{g}}(\boldsymbol{\alpha})$ is upper bounded as: where $\boldsymbol{\Theta} = [\boldsymbol{\theta}_1,\boldsymbol{\theta}_2,\dots,\boldsymbol{\theta}_K]$ and $\boldsymbol{\theta}=\boldsymbol{\Theta}\boldsymbol{\alpha}$.

Figures (1)

  • Figure 1: Fine-tuning test accuracy vs. lightweight finetuning epochs (with fixed $\boldsymbol{\alpha}$). Datasets: (a) CIFAR-10, (b) SVHN, (c) Imagenette. We compare Configs 1-4 (see Sec. \ref{['sec:experiments']}). GLUE (Config 4; ours) shows similar trends to Config 3 and consistently achieves higher test accuracy and faster convergence than Configs 1-2.

Theorems & Definitions (2)

  • Proposition 1: Variance Bound
  • proof