Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Designing a Robust, Bounded, and Smooth Loss Function for Improved Supervised Learning

Soumi Mahato, Lineesh M. C

TL;DR

The paper introduces RoBoS-NN, a robust, bounded, and smooth regression loss obtained by extending the RoBoSS classifier loss to continuous outputs. The loss is defined as $L_{RoBoS-NN}(u) = \lambda \{ 1 - ( a \sqrt{u^{2}+\epsilon} - a \sqrt{\epsilon} ) \exp(- ( a \sqrt{u^{2}+\epsilon} - a \sqrt{\epsilon} )) \}$ with $u = |y - \hat{y}|$, and the neural-network objective is $\min_{\theta} \frac{\lambda}{2} \|\theta\|^{2} + \frac{1}{n} \sum_{k=1}^{n} L_{RoBoS-NN}(y_k - f(x_k; \theta))$. A theoretical generalization bound based on Rademacher complexity is derived, showing the bound scales with the Lipschitz-like constant $a/e$ and network norms. Empirically, RoBoS-NN is evaluated on four real-world time-series datasets with injected outliers and outperforms standard losses (MAE, MSE, Huber, Logcosh) in robustness and accuracy, with hyperparameters tuned via HyperOpt/TPE. The results suggest RoBoS-NN is ready to be integrated into broader architectures (e.g., CNNs, RNNs) to enhance robustness in supervised learning and forecasting tasks.

Abstract

The loss function is crucial to machine learning, especially in supervised learning frameworks. It is a fundamental component that controls the behavior and general efficacy of learning algorithms. However, despite their widespread use, traditional loss functions have significant drawbacks when dealing with high-dimensional and outlier-sensitive datasets, which frequently results in reduced performance and slower convergence during training. In this work, we develop a robust, bounded, and smooth (RoBoS-NN) loss function to resolve the aforementioned hindrances. The generalization ability of the loss function has also been theoretically analyzed to rigorously justify its robustness. Moreover, we implement RoboS-NN loss in the framework of a neural network (NN) to forecast time series and present a new robust algorithm named $\mathcal{L}_{\text{RoBoS}}$-NN. To assess the potential of $\mathcal{L}_{\text{RoBoS}}$-NN, we conduct experiments on multiple real-world datasets. In addition, we infuse outliers into data sets to evaluate the performance of $\mathcal{L}_{\text{RoBoS}}$-NN in more challenging scenarios. Numerical results show that $\mathcal{L}_{\text{RoBoS}}$-NN outperforms the other benchmark models in terms of accuracy measures.

Designing a Robust, Bounded, and Smooth Loss Function for Improved Supervised Learning

TL;DR

The paper introduces RoBoS-NN, a robust, bounded, and smooth regression loss obtained by extending the RoBoSS classifier loss to continuous outputs. The loss is defined as with , and the neural-network objective is . A theoretical generalization bound based on Rademacher complexity is derived, showing the bound scales with the Lipschitz-like constant and network norms. Empirically, RoBoS-NN is evaluated on four real-world time-series datasets with injected outliers and outperforms standard losses (MAE, MSE, Huber, Logcosh) in robustness and accuracy, with hyperparameters tuned via HyperOpt/TPE. The results suggest RoBoS-NN is ready to be integrated into broader architectures (e.g., CNNs, RNNs) to enhance robustness in supervised learning and forecasting tasks.

Abstract

The loss function is crucial to machine learning, especially in supervised learning frameworks. It is a fundamental component that controls the behavior and general efficacy of learning algorithms. However, despite their widespread use, traditional loss functions have significant drawbacks when dealing with high-dimensional and outlier-sensitive datasets, which frequently results in reduced performance and slower convergence during training. In this work, we develop a robust, bounded, and smooth (RoBoS-NN) loss function to resolve the aforementioned hindrances. The generalization ability of the loss function has also been theoretically analyzed to rigorously justify its robustness. Moreover, we implement RoboS-NN loss in the framework of a neural network (NN) to forecast time series and present a new robust algorithm named -NN. To assess the potential of -NN, we conduct experiments on multiple real-world datasets. In addition, we infuse outliers into data sets to evaluate the performance of -NN in more challenging scenarios. Numerical results show that -NN outperforms the other benchmark models in terms of accuracy measures.
Paper Structure (10 sections, 1 theorem, 14 equations, 2 figures, 4 tables, 1 algorithm)

This paper contains 10 sections, 1 theorem, 14 equations, 2 figures, 4 tables, 1 algorithm.

Table of Contents

  1. Introduction and Motivation
  2. Proposed work
  3. THEORETICAL EVALUATION OF THE PROPOSED ROBOS_NN LOSS FUNCTION
  4. Optimization of $\mathcal{L}_{\text{RoBoS}}$-NN
  5. Computational Analysis
  6. Experimental Results
  7. Numerical experiments and Result Analysis
  8. Evaluation on clean and outlier-contaminated time series datasets
  9. Conclusion and Future Work
  10. Acknowledgments

Key Result

Theorem 1

Given, ${H_d}$ be the class of real-valued networks of depth $d$ over the domain $X$, where each parameter matrix $W_j$ has Frobenius norm atmost $M_F(j)$ and with $1$-Lipschitz, positive homogeneous activation functions. Let $f_p$ be the predictor produced by Robos-NN. Then for any, $0<\epsilon<1$,

Figures (2)

  • Figure 1: Comparison of loss function profiles: (a) MAE loss, (b) MSE loss, (c) Huber loss, (d) Log-cosh loss, (e) RoBoS-NN loss with varying shape parameter $\lambda$, (f) RoBoS-NN loss with varying robustness parameter $a$.
  • Figure 2: Training framework of the proposed neural-network-based forecasting model using the RoBoS loss optimized via the Adam algorithm.

Theorems & Definitions (2)

  • Theorem 1
  • proof