Neural Loss Function Evolution for Large-Scale Image Classifier Convolutional Neural Networks

TL;DR

Three new loss functions are discovered, called the NeuroLoss functions, that were able to outperform cross-entropy in terms of a higher mean test accuracy as a simple drop-in replacement loss function across the majority of experiments.

Abstract

For classification, neural networks typically learn by minimizing cross-entropy, but are evaluated and compared using accuracy. This disparity suggests neural loss function search (NLFS), the search for a drop-in replacement loss function of cross-entropy for neural networks. We apply NLFS to image classifier convolutional neural networks. We propose a new search space for NLFS that encourages more diverse loss functions to be explored, and a surrogate function that accurately transfers to large-scale convolutional neural networks. We search the space using regularized evolution, a mutation-only aging genetic algorithm. After evolution and a proposed loss function elimination protocol, we transferred the final loss functions across multiple architectures, datasets, and image augmentation techniques to assess generalization. In the end, we discovered three new loss functions, called NeuroLoss1, NeuroLoss2, and NeuroLoss3 that were able to outperform cross-entropy in terms of a higher mean test accuracy as a simple drop-in replacement loss function across the majority of experiments.

Figures (4)

• Figure 1: Example loss function with three active (blue) hidden state nodes, one inactive hidden node (grey), and one root node (white). The final loss equation is given next to the root node in the box.
• Figure 2: Overview of the binary phenotypes of cross-entropy, NeuroLoss 1, NeuroLoss 2, and NeuroLoss 3.
• Figure 3: Zoomed in view (around $\hat{y}\to 1$) of the binary phenotypes of cross-entropy, NeuroLoss 1, NeuroLoss 2, and NeuroLoss 3.
• Figure 4: Results for the evolutionary algorithm evaluated using the proposed surrogate function.