Exploring the Efficacy of Meta-Learning: Unveiling Superior Data Diversity Utilization of MAML Over Pre-training

TL;DR

This work challenges the prevailing focus on data size by probing dataset diversity as a key driver of vision-model performance. Using Task2Vec to quantify diversity, it compares pre-training and Model-Agnostic Meta-Learning (MAML) variants across twelve visual datasets, uncovering positive correlations between diversity and accuracy, with higher-order MAML showing the strongest dependence ($R^2$ up to $0.416$ for accuracy). The findings suggest data diversity as a critical, actionable factor that can enhance generalization and potentially reduce the computational burden of scaling. The study advocates a shift toward quality-aware data selection to complement scaling in building more powerful and efficient vision models.

Abstract

Currently, data and model size dominate the narrative in the training of super-large, powerful models. However, there has been a lack of exploration on the effect of other attributes of the training dataset on model performance. We hypothesize that dataset diversity can impact the performance of vision models. Our study shows positive correlations between test set accuracy and data diversity, providing an argument for furthering the research of dataset attributes beyond size. We analyzed pre-training and model-agnostic meta-learning methods on twelve popular visual datasets (e.g., Omniglot, CIFAR-FS, Aircraft) and five model configurations, including MAML variants with different numbers of inner gradient steps and supervised learning. We show moderate to strong positive correlations (R-squared: 0.15-0.42) between accuracy and data diversity and weaker but significant correlations (R-squared: ~0.2) between loss and diversity. These findings support our hypothesis and demonstrate a promising way for a deeper exploration of how formal data diversity influences model performance. This initial study highlights the potential of (Task2Vec) data diversity as a valuable measure in the rapidly evolving field of large-scale learning and emphasizes that understanding the dataset is key to building more powerful and generalizable models.

Figures (1)

• Figure 1: We validate that training on more formally diverse data leads to better performance across different settings. We do this by showing moderate to strong positive correlations between accuracy and data diversity in HO MAML models and weaker but significant correlations in FO MAML, PT models. HO MAML stands for Higher-Order Model Agnostic Meta-Learning, FO stands for First-Order Model Agnostic Meta-Learning, and PT for Pre-training.