Unleashing the Power of Meta-tuning for Few-shot Generalization Through Sparse Interpolated Experts

TL;DR

This paper tackles the problem of improving few-shot generalization for vision foundation models, especially under distribution shifts. It proposes Sparse MetA-Tuning (SMAT), a method that learns a pool of sparse experts and a gating mechanism to interpolate between pre-trained weights and task-specific adaptations, added to a frozen backbone. SMAT uses a hypernetwork to generate per-task expert weights and employs sparsity-aware masks with a KD objective to promote specialization, achieving state-of-the-art results on Meta-Dataset with added OOD tasks and demonstrating strong ID/OOD balance. The approach is compatible with gradient-based and gradient-free meta-testing and with parameter-efficient fine-tuning, and analyses reveal interpretable, task-related sparsity patterns that adapt to distribution shifts and reduce meta-overfitting, with broad practical implications for robust transfer learning.

Abstract

Recent successes suggest that parameter-efficient fine-tuning of foundation models as the state-of-the-art method for transfer learning in vision, replacing the rich literature of alternatives such as meta-learning. In trying to harness the best of both worlds, meta-tuning introduces a subsequent optimization stage of foundation models but has so far only shown limited success and crucially tends to underperform on out-of-distribution (OOD) tasks. In this paper, we introduce Sparse MetA-Tuning (SMAT), a method inspired by sparse mixture-of-experts approaches and trained to isolate subsets of pre-trained parameters automatically for meta-tuning on each task. SMAT successfully overcomes OOD sensitivity and delivers on the promise of enhancing the transfer abilities of vision foundation models beyond parameter-efficient fine-tuning. We establish new state-of-the-art results on a challenging combination of Meta-Dataset augmented with additional OOD tasks in both zero-shot and gradient-based adaptation settings. In addition, we provide a thorough analysis of the superiority of learned over hand-designed sparsity patterns for sparse expert methods and the pivotal importance of the sparsity level in balancing between in-distribution and out-of-distribution generalization. Our code is publicly available.

Figures (7)

• Figure 1: Average testing accuracy during meta-training for meta-tuning methods. SMAT yields better ID and OOD results and shows an attractive learning speedup.
• Figure 2: Overview of the proposed Sparse MetA-Tuning approach, showing meta-training and inference procedures for a single task ${\mathcal{T}}_i$. SMAT meta-learns a shared knowledge pool $\mathcal{M}$ consisting of $|\mathcal{M}|$ sparse interpolated experts characterized by a common, learnable set of dense parameters $\boldsymbol{\theta}^\delta$ and distinct, learnable sets of gating masks $\{\boldsymbol{z}_m\}^{|\mathcal{M}|}_{m=1}$ with sparsity constraints. To construct each task-specific model $\boldsymbol{\theta}_i$ for both meta-training and inference, (1) SMAT first combines experts via a weighted-sum with merging weights $\boldsymbol{\alpha}_i$ generated by a meta-learned hypernetwork $h_{\boldsymbol{\zeta}}$ based on the task's support set $\mathcal{T}_i^s$. (2) The experts are then subsequently combined with the frozen pre-trained model $\boldsymbol{\theta}^\mathtt{pre}$ to enhance both in-distribution (ID) and out-of-distribution (OOD) generalization performance. Alongside (3) the query prediction loss ${\mathcal{L}}^\mathtt{ce}_{\mathcal{T}_i^q}(\boldsymbol{\theta}_i)$, (4) knowledge distillation with task-specific dense teachers ${\mathcal{L}}^\mathtt{kd}_{\mathcal{T}_i^q}(\boldsymbol{\theta}_i,\boldsymbol{\theta}_i^\mathtt{tr})$ is introduced during meta-training to promote specialization and cooperation of the sparse interpolated experts, ensuring optimization success.
• Figure 3: Average performance tradeoff on sampled ID vs OOD tasks as a function of (color) expert sparsity level $\tau$, and (marker) number of experts.
• Figure 4: Meta-gradients alignment between tasks throughout for SMAT with low and high sparsity levels. Meta-gradients are calculated w.r.t. the parameters shown in the legend.
• Figure 5: (a-b) Meta-learned model capacity after meta-tuning (i.e., number of non-zero parameters) grouped by a): layer types, and b): layer depth. (c-d) Expert specialisation. c) Dendrogram of task similarity based on expert selection scores. d) Overlap between masks.