Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

B-SMALL: A Bayesian Neural Network approach to Sparse Model-Agnostic Meta-Learning

Anish Madan, Ranjitha Prasad

TL;DR

The proposed framework incorporates a sparse variational loss term alongside the loss function of MAML, which uses a sparsifying approximated KL divergence as a regularizer, and demonstrates applicability of the approach in distributed sensor networks, where sparsity and meta-learning can be beneficial.

Abstract

There is a growing interest in the learning-to-learn paradigm, also known as meta-learning, where models infer on new tasks using a few training examples. Recently, meta-learning based methods have been widely used in few-shot classification, regression, reinforcement learning, and domain adaptation. The model-agnostic meta-learning (MAML) algorithm is a well-known algorithm that obtains model parameter initialization at meta-training phase. In the meta-test phase, this initialization is rapidly adapted to new tasks by using gradient descent. However, meta-learning models are prone to overfitting since there are insufficient training tasks resulting in over-parameterized models with poor generalization performance for unseen tasks. In this paper, we propose a Bayesian neural network based MAML algorithm, which we refer to as the B-SMALL algorithm. The proposed framework incorporates a sparse variational loss term alongside the loss function of MAML, which uses a sparsifying approximated KL divergence as a regularizer. We demonstrate the performance of B-MAML using classification and regression tasks, and highlight that training a sparsifying BNN using MAML indeed improves the parameter footprint of the model while performing at par or even outperforming the MAML approach. We also illustrate applicability of our approach in distributed sensor networks, where sparsity and meta-learning can be beneficial.

B-SMALL: A Bayesian Neural Network approach to Sparse Model-Agnostic Meta-Learning

TL;DR

The proposed framework incorporates a sparse variational loss term alongside the loss function of MAML, which uses a sparsifying approximated KL divergence as a regularizer, and demonstrates applicability of the approach in distributed sensor networks, where sparsity and meta-learning can be beneficial.

Abstract

There is a growing interest in the learning-to-learn paradigm, also known as meta-learning, where models infer on new tasks using a few training examples. Recently, meta-learning based methods have been widely used in few-shot classification, regression, reinforcement learning, and domain adaptation. The model-agnostic meta-learning (MAML) algorithm is a well-known algorithm that obtains model parameter initialization at meta-training phase. In the meta-test phase, this initialization is rapidly adapted to new tasks by using gradient descent. However, meta-learning models are prone to overfitting since there are insufficient training tasks resulting in over-parameterized models with poor generalization performance for unseen tasks. In this paper, we propose a Bayesian neural network based MAML algorithm, which we refer to as the B-SMALL algorithm. The proposed framework incorporates a sparse variational loss term alongside the loss function of MAML, which uses a sparsifying approximated KL divergence as a regularizer. We demonstrate the performance of B-MAML using classification and regression tasks, and highlight that training a sparsifying BNN using MAML indeed improves the parameter footprint of the model while performing at par or even outperforming the MAML approach. We also illustrate applicability of our approach in distributed sensor networks, where sparsity and meta-learning can be beneficial.

Paper Structure

This paper contains 10 sections, 4 equations, 2 figures, 3 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Preliminaries
  3. Model-Agnostic Meta-Learning (MAML)
  4. Bayesian Neural Networks and Sparsity
  5. BNN based Sparse MAML (B-SMALL)
  6. Experiments and Results
  7. $K$-Shot Regression
  8. Few-Shot Classification
  9. Use-case: Sparse MAML in Sensor Networks
  10. Conclusion and Future Work

Figures (2)

  • Figure 1: Plot of MSE vs Number of Gradient Steps taken at meta-test time for $K=10$ Sinusoid Regression.
  • Figure 2: Overfitting in the case of MAML and B-SMALL: Note that difference between train and validation loss for MAML is much higher than that for B-SMALL, thereby showing the effect of regularization and enabling better learning.