XB-MAML: Learning Expandable Basis Parameters for Effective Meta-Learning with Wide Task Coverage

TL;DR

XB-MAML tackles the challenge of meta-learning across wide task distributions by learning expandable basis initializations that are linearly combined to form task-specific starts. Expansion is guided by a projection-error metric $\epsilon$ that signals when current bases fail to cover new tasks, with new bases sampled from a Gaussian distribution to grow the basis set. The approach yields state-of-the-art results on multi-domain benchmarks and demonstrates robust cross-domain transfer, supported by analyses of basis coverage, expansion dynamics, and computational efficiency. By treating initializations as a growth-capable basis and enforcing orthogonality, XB-MAML broadens the inductive biases available for unseen tasks, enabling more effective rapid adaptation in diverse domains.

Abstract

Meta-learning, which pursues an effective initialization model, has emerged as a promising approach to handling unseen tasks. However, a limitation remains to be evident when a meta-learner tries to encompass a wide range of task distribution, e.g., learning across distinctive datasets or domains. Recently, a group of works has attempted to employ multiple model initializations to cover widely-ranging tasks, but they are limited in adaptively expanding initializations. We introduce XB-MAML, which learns expandable basis parameters, where they are linearly combined to form an effective initialization to a given task. XB-MAML observes the discrepancy between the vector space spanned by the basis and fine-tuned parameters to decide whether to expand the basis. Our method surpasses the existing works in the multi-domain meta-learning benchmarks and opens up new chances of meta-learning for obtaining the diverse inductive bias that can be combined to stretch toward the effective initialization for diverse unseen tasks.

Figures (7)

• Figure 1: Illustration of bi-level optimization with three initializations $\{\theta^{(m)}\}_{m=1}^{3}$. (a) TSA-MAML selects one initialization with the smallest loss. (b) MUSML separately fine-tunes and meta-updates each initialization. (c) XB-MAML forms the initialization $\theta^{\star}$ via linear combination and jointly meta-updates them.
• Figure 2: The conceptual illustrations of XB-MAML: (a) outlines the XB-MAML learning process. (b) illustrates the visual representation of the scenario that XB-MAML expands extra basis.
• Figure 3: Visualization results in Meta-Datasets-ABF. (a) illustrates a t-SNE plot comparing the finetuned parameters and initial model parameters. (b) represents the average normalized coefficients $\sigma$ indicating the impact of each $\theta^{(m)}$ when linearly combined with the datasets.
• Figure 4: $\epsilon$ on Meta-Datasets-BTAF
• Figure 5: Normalized singular values of initializations