Meta-Learning with Heterogeneous Tasks

TL;DR

HeTRoM is proficient in handling heterogeneous tasks, and it prevents easy tasks from overwhelming the meta-learner, and it allows for an efficient iterative optimization algorithm based on bi-level optimization, which is then improved by integrating statistical guidance.

Abstract

Meta-learning is a general approach to equip machine learning models with the ability to handle few-shot scenarios when dealing with many tasks. Most existing meta-learning methods work based on the assumption that all tasks are of equal importance. However, real-world applications often present heterogeneous tasks characterized by varying difficulty levels, noise in training samples, or being distinctively different from most other tasks. In this paper, we introduce a novel meta-learning method designed to effectively manage such heterogeneous tasks by employing rank-based task-level learning objectives, Heterogeneous Tasks Robust Meta-learning (HeTRoM). HeTRoM is proficient in handling heterogeneous tasks, and it prevents easy tasks from overwhelming the meta-learner. The approach allows for an efficient iterative optimization algorithm based on bi-level optimization, which is then improved by integrating statistical guidance. Our experimental results demonstrate that our method provides flexibility, enabling users to adapt to diverse task settings and enhancing the meta-learner's overall performance.

Figures (4)

• Figure 1: Heterogeneous tasks in meta-learning for image retrieval on real-world datasets (Mini-ImageNet and Tiered-ImageNet). (Top) Three different types of tasks based on their difficulty. (Bottom) Histograms of losses can be used to differentiate three types of tasks. The displayed histogram is derived from real-world datasets, the details are provided in texts.
• Figure 2: Behavior analysis of meta-learning. (a) The loss distributions of training tasks with models trained with all tasks and hard tasks, respectively. (b) The loss distributions of clean and noisy tasks with models trained with a clean dataset ("Clean training") and a noisy dataset ("Noisy training"), respectively. (c) The frequency of each task index with the highest loss throughout the training process, where the outlier task is assigned index 15 in each iteration. "Test accuracy" (%) is obtained by testing on a separate clean test set.
• Figure 3: (a) Frequency of the noise ratio for each task in a mini-batch being excluded by $\hat{\lambda}$ during meta-training. (b) Loss distribution of clean and outlier tasks when conducting fast adaptation with a trained meta-model. (c) Test accuracy with varying $k$ and $m$ in clean task setting. (d) Test accuracy with varying $k$ and $m$ in fixed noisy task setting.
• Figure B.1: Illustration of statistic-guided learning for solving HeTRoM.

Theorems & Definitions (5)

• Theorem 1
• Lemma 1
• Proof 1
• Theorem B.1
• Proof 2