Learning Symbolic Model-Agnostic Loss Functions via Meta-Learning

TL;DR

The paper addresses the challenge of designing effective, task- and model-agnostic loss functions for neural networks. It proposes EvoMAL, a hybrid neuro-symbolic framework that uses genetic programming to discover symbolic loss functions and unrolled differentiation to optimize their coefficients, resulting in a computationally tractable yet expressive approach. The method is evaluated offline across diverse datasets and architectures, showing that meta-learned symbolic losses can outperform handcrafted losses and prior loss-learning methods, with insights into loss structures, transferability, and implicit learning-rate effects. This work advances loss-function learning by delivering interpretable, transferable losses and demonstrates practical gains without altering standard training pipelines.

Abstract

In this paper, we develop upon the emerging topic of loss function learning, which aims to learn loss functions that significantly improve the performance of the models trained under them. Specifically, we propose a new meta-learning framework for learning model-agnostic loss functions via a hybrid neuro-symbolic search approach. The framework first uses evolution-based methods to search the space of primitive mathematical operations to find a set of symbolic loss functions. Second, the set of learned loss functions are subsequently parameterized and optimized via an end-to-end gradient-based training procedure. The versatility of the proposed framework is empirically validated on a diverse set of supervised learning tasks. Results show that the meta-learned loss functions discovered by the newly proposed method outperform both the cross-entropy loss and state-of-the-art loss function learning methods on a diverse range of neural network architectures and datasets.

Figures (19)

• Figure 1: An overview of the EvoMAL framework, showing the pipeline for learning loss functions at meta-training time.
• Figure 2: Overview of the transitional procedure used to covert $\mathcal{M}$ into a trainable meta-loss network $\mathcal{M}_{\phi}^{\mathsf{T}}$.
• Figure 3: Overview of the EvoMAL algorithm at meta-testing time, where the base network $f_{\theta}(x)$, shown (left) as the popular LeNet-5 architecture, is trained using the meta-loss network $\mathcal{M}_{\phi}^{\mathsf{T}}$ (right) found at meta-training time as the loss function.
• Figure 4: Mean meta-training learning curves across 5 independent executions of each algorithm, showing the fitness score (y-axis) against generations (x-axis), where a generation is equivalent to 25 evaluations. Best viewed in color.
• Figure 5: Frequencies of occurrence of the time saving filters, and the corresponding frequencies of loss optimization and evaluation throughout the symbolic search process. Reporting the average frequencies across all task + model pairs and independent executions of EvoMAL.