Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Alternate Loss Functions for Classification and Robust Regression Can Improve the Accuracy of Artificial Neural Networks

Mathew Mithra Noel, Arindam Banerjee, Yug Oswal, Geraldine Bessie Amali D, Venkataraman Muthiah-Nakarajan

TL;DR

This paper explores the effect on performance of using new loss functions that are also convex but penalize errors differently compared to the popular Cross-entropy loss.

Abstract

All machine learning algorithms use a loss, cost, utility or reward function to encode the learning objective and oversee the learning process. This function that supervises learning is a frequently unrecognized hyperparameter that determines how incorrect outputs are penalized and can be tuned to improve performance. This paper shows that training speed and final accuracy of neural networks can significantly depend on the loss function used to train neural networks. In particular derivative values can be significantly different with different loss functions leading to significantly different performance after gradient descent based Backpropagation (BP) training. This paper explores the effect on performance of using new loss functions that are also convex but penalize errors differently compared to the popular Cross-entropy loss. Two new classification loss functions that significantly improve performance on a wide variety of benchmark tasks are proposed. A new loss function call smooth absolute error that outperforms the Squared error, Huber and Log-Cosh losses on datasets with significantly many outliers is proposed. This smooth absolute error loss function is infinitely differentiable and more closely approximates the absolute error loss compared to the Huber and Log-Cosh losses used for robust regression.

Alternate Loss Functions for Classification and Robust Regression Can Improve the Accuracy of Artificial Neural Networks

TL;DR

This paper explores the effect on performance of using new loss functions that are also convex but penalize errors differently compared to the popular Cross-entropy loss.

Abstract

All machine learning algorithms use a loss, cost, utility or reward function to encode the learning objective and oversee the learning process. This function that supervises learning is a frequently unrecognized hyperparameter that determines how incorrect outputs are penalized and can be tuned to improve performance. This paper shows that training speed and final accuracy of neural networks can significantly depend on the loss function used to train neural networks. In particular derivative values can be significantly different with different loss functions leading to significantly different performance after gradient descent based Backpropagation (BP) training. This paper explores the effect on performance of using new loss functions that are also convex but penalize errors differently compared to the popular Cross-entropy loss. Two new classification loss functions that significantly improve performance on a wide variety of benchmark tasks are proposed. A new loss function call smooth absolute error that outperforms the Squared error, Huber and Log-Cosh losses on datasets with significantly many outliers is proposed. This smooth absolute error loss function is infinitely differentiable and more closely approximates the absolute error loss compared to the Huber and Log-Cosh losses used for robust regression.
Paper Structure (11 sections, 10 equations, 8 figures, 5 tables)

This paper contains 11 sections, 10 equations, 8 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Mathematical Formalism
  3. New loss functions for classifciation
  4. New loss functions for robust regression
  5. Comparison of performance on benchmark vision datasets
  6. Comparison on the CIFAR-10 benchmark
  7. Comparison on Imagenette Dataset
  8. Comparison on text datasets
  9. Comparison on Consumer Financial Protection Bureau (CFPB) dataset
  10. Comparison on the IMDB Movie Review benchmark
  11. Conclusion

Figures (8)

  • Figure 1: Plot of Binary Cross-entropy loss when the target $y = 0$ (left) and $y = 1$ (right).
  • Figure 2: Plot of different loss functions when the target y = 1.
  • Figure 3: Plot of different loss functions used for robust regression.
  • Figure 4: Effect of using different loss functions for linear regression when the dataset has significant number of outliers. The training dataset is shown as black dots and consists of 1500 instances with 500 outliers.
  • Figure 5: Variation of accuracy with epochs on the CIFAR-10 benchmark.
  • ...and 3 more figures