MAC: A Meta-Learning Approach for Feature Learning and Recombination

TL;DR

MAC extends optimization-based meta-learning by introducing Additional Connection Units (ACUs) to widen the network, enabling learning of new task-specific features during meta-testing while freezing base meta-trained parameters to preserve prior knowledge. By combining ACU-driven feature learning with existing meta-learned representations, MAC achieves improved adaptation under distribution shifts, outperforming ANIL on perturbed distributions in few-shot tasks (notably around $\approx 13\%$ in 5-shot scenarios). The approach emphasizes width over depth for accommodating new features and provides empirical guidance on ACU quantities, placement, and depth, along with an analysis of computational implications. Overall, MAC demonstrates that controlled feature learning and recombination are crucial for robust meta-testing in non-identical task distributions, offering a practical route for few-shot learning under real-world distribution shifts.

Abstract

Optimization-based meta-learning aims to learn an initialization so that a new unseen task can be learned within a few gradient updates. Model Agnostic Meta-Learning (MAML) is a benchmark algorithm comprising two optimization loops. The inner loop is dedicated to learning a new task and the outer loop leads to meta-initialization. However, ANIL (almost no inner loop) algorithm shows that feature reuse is an alternative to rapid learning in MAML. Thus, the meta-initialization phase makes MAML primed for feature reuse and obviates the need for rapid learning. Contrary to ANIL, we hypothesize that there may be a need to learn new features during meta-testing. A new unseen task from non-similar distribution would necessitate rapid learning in addition reuse and recombination of existing features. In this paper, we invoke the width-depth duality of neural networks, wherein, we increase the width of the network by adding extra computational units (ACU). The ACUs enable the learning of new atomic features in the meta-testing task, and the associated increased width facilitates information propagation in the forwarding pass. The newly learnt features combine with existing features in the last layer for meta-learning. Experimental results show that our proposed MAC method outperformed existing ANIL algorithm for non-similar task distribution by approximately 13% (5-shot task setting)

Figures (4)

• Figure 1: This diagram shows the proposed MAC model $M'$. The neural network inputs a batch of task and modifies the weights of the networks in each iterations. Each convolution layer has blue and green filters. The blue filters are the base modules and green filters are the newly added units(ACUs). Blue dashed lines are the old connection of the base model M, Whereas green dotted connections are the new links with the old and new nodes. Solid black lines are zero weight connections.
• Figure 2: Part (a) illustrates the meta-training phase of our proposed MAC algorithm where it finds meta-initialization parameters $\theta^*$ and $\mu^*$ after the final outer loop updation for the model $M'$ so that new tasks are learnt rapidly with additional feature learning and recombination, part (b) Represents model $M'$ with additional links to the [L-1] layers shown as ${\theta^+}$ and the classifier layer weights as $\mu^*$ obtained during meta-training to perform task adaptation.
• Figure 3: Three sets of different character depicting three classes from Omniglot dataset. ${char1}$, $char4$ and $char16$ in Hebrew. Each character is written by $20$ different person.
• Figure 4: 5-way 5-shot test accuracy graph on omniglot dataset.