Fast and Efficient Local Search for Genetic Programming Based Loss Function Learning

TL;DR

A new meta-learning framework for task and model-agnostic loss function learning via a hybrid search approach that uses genetic programming to find a set of symbolic loss functions and is subsequently parameterized and optimized via unrolled differentiation.

Abstract

In this paper, we develop upon the topic of loss function learning, an emergent meta-learning paradigm that aims to learn loss functions that significantly improve the performance of the models trained under them. Specifically, we propose a new meta-learning framework for task and model-agnostic loss function learning via a hybrid search approach. The framework first uses genetic programming to find a set of symbolic loss functions. Second, the set of learned loss functions is subsequently parameterized and optimized via unrolled differentiation. The versatility and performance of the proposed framework are empirically validated on a diverse set of supervised learning tasks. Results show that the learned loss functions bring improved convergence, sample efficiency, and inference performance on tabulated, computer vision, and natural language processing problems, using a variety of task-specific neural network architectures.

Figures (6)

• Figure 1: Overview of the EvoMAL algorithm.
• Figure 2: Overview of the constraint enforcement procedure, where (a) is a constraint violating expression, (b) demonstrates enforcing the required arguments constraint, and (c) shows enforcing the non-negative output constraint.
• Figure 3: Overview of the transitional procedure used to covert $\mathcal{M}$ into a trainable meta-loss network $\mathcal{M}_{\phi}^{\mathsf{T}}$.
• Figure 4: Example of one step of the inner loss optimization process used in EvoMAL to learn the weights $\phi$ of the meta-loss network $\mathcal{M}_{\phi}^{\mathsf{T}}$ with respect to the base network $f_{\theta}(x)$ shown (left) as the popular LeNet-5 architecture at meta-training time.
• Figure 5: Mean meta-testing learning curves on the out-of-sample testing tasks across 10 independent executions of each algorithm, showing the performance (y-axis) against gradient steps (x-axis). Best viewed in color.