Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Perturbing the Gradient for Alleviating Meta Overfitting

Manas Gogoi, Sambhavi Tiwari, Shekhar Verma

TL;DR

A number of solutions to tackle meta-overfitting on few-shot learning settings, such as few-shot sinusoid regression and few shot classification are proposed, which demonstrate improved generalization performance compared to state-of-the-art baselines for learning in a non-mutually exclusive task setting.

Abstract

The reason for Meta Overfitting can be attributed to two factors: Mutual Non-exclusivity and the Lack of diversity, consequent to which a single global function can fit the support set data of all the meta-training tasks and fail to generalize to new unseen tasks. This issue is evidenced by low error rates on the meta-training tasks, but high error rates on new tasks. However, there can be a number of novel solutions to this problem keeping in mind any of the two objectives to be attained, i.e. to increase diversity in the tasks and to reduce the confidence of the model for some of the tasks. In light of the above, this paper proposes a number of solutions to tackle meta-overfitting on few-shot learning settings, such as few-shot sinusoid regression and few shot classification. Our proposed approaches demonstrate improved generalization performance compared to state-of-the-art baselines for learning in a non-mutually exclusive task setting. Overall, this paper aims to provide insights into tackling overfitting in meta-learning and to advance the field towards more robust and generalizable models.

Perturbing the Gradient for Alleviating Meta Overfitting

TL;DR

A number of solutions to tackle meta-overfitting on few-shot learning settings, such as few-shot sinusoid regression and few shot classification are proposed, which demonstrate improved generalization performance compared to state-of-the-art baselines for learning in a non-mutually exclusive task setting.

Abstract

The reason for Meta Overfitting can be attributed to two factors: Mutual Non-exclusivity and the Lack of diversity, consequent to which a single global function can fit the support set data of all the meta-training tasks and fail to generalize to new unseen tasks. This issue is evidenced by low error rates on the meta-training tasks, but high error rates on new tasks. However, there can be a number of novel solutions to this problem keeping in mind any of the two objectives to be attained, i.e. to increase diversity in the tasks and to reduce the confidence of the model for some of the tasks. In light of the above, this paper proposes a number of solutions to tackle meta-overfitting on few-shot learning settings, such as few-shot sinusoid regression and few shot classification. Our proposed approaches demonstrate improved generalization performance compared to state-of-the-art baselines for learning in a non-mutually exclusive task setting. Overall, this paper aims to provide insights into tackling overfitting in meta-learning and to advance the field towards more robust and generalizable models.
Paper Structure (14 sections, 3 equations, 3 figures, 5 tables, 2 algorithms)

This paper contains 14 sections, 3 equations, 3 figures, 5 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Problem Definition
  3. Related Work
  4. Preliminaries
  5. Meta Learning
  6. Non-Mutual Exclusivity
  7. Proposed Methodology
  8. Feedback from Meta-Test to Meta-Train
  9. Experiments and Results
  10. Model Architecture
  11. Sinusoid Regression
  12. Classification on Omniglot, MiniImageNet and D'Claw
  13. Effect of noise addition on the accuracy :
  14. Conclusion

Figures (3)

  • Figure 1: Schematic diagram of the proposed approach. Under the condition of non-mutually exclusive tasks, the optimization process (shown in solid lines) leads the trajectory to the minima in space that meta-overfits or memorizes. However, with the addition of noise, as in our approach, the optimization trajectory (shown in dotted lines) probabilistically arrives at some other point of minima that does not meta-overfit
  • Figure 2: MSE Loss for Sinusoid regression tasks under the non-mutually exclusive task setting averaged over 10 trials.
  • Figure 3: Variation of accuracy for 5 way 1 shot Omniglot dataset classification with varying standard deviation of the noise distribution