HawkEye: Advancing Robust Regression with Bounded, Smooth, and Insensitive Loss Function

TL;DR

This paper introduces a novel symmetric loss function named the HawkEye loss function, which stands out as the first loss function in SVR literature to be bounded, smooth, and simultaneously possess an insensitive zone, and proposes a new fast and robust model termed HE-LSSVR.

Abstract

Support vector regression (SVR) has garnered significant popularity over the past two decades owing to its wide range of applications across various fields. Despite its versatility, SVR encounters challenges when confronted with outliers and noise, primarily due to the use of the $\varepsilon$-insensitive loss function. To address this limitation, SVR with bounded loss functions has emerged as an appealing alternative, offering enhanced generalization performance and robustness. Notably, recent developments focus on designing bounded loss functions with smooth characteristics, facilitating the adoption of gradient-based optimization algorithms. However, it's crucial to highlight that these bounded and smooth loss functions do not possess an insensitive zone. In this paper, we address the aforementioned constraints by introducing a novel symmetric loss function named the HawkEye loss function. It is worth noting that the HawkEye loss function stands out as the first loss function in SVR literature to be bounded, smooth, and simultaneously possess an insensitive zone. Leveraging this breakthrough, we integrate the HawkEye loss function into the least squares framework of SVR and yield a new fast and robust model termed HE-LSSVR. The optimization problem inherent to HE-LSSVR is addressed by harnessing the adaptive moment estimation (Adam) algorithm, known for its adaptive learning rate and efficacy in handling large-scale problems. To our knowledge, this is the first time Adam has been employed to solve an SVR problem. To empirically validate the proposed HE-LSSVR model, we evaluate it on UCI, synthetic, and time series datasets. The experimental outcomes unequivocally reveal the superiority of the HE-LSSVR model both in terms of its remarkable generalization performance and its efficiency in training time.

Figures (4)

• Figure 1: Visual illustration of some baseline loss functions. (a) Least square loss function. (b) $\varepsilon$-insensitive loss function with $\varepsilon=0.5$. (c) Huber loss function with $\theta=5$. (d) Canal loss function with $\varepsilon=0.5$ and $\theta=1.5$. (e) Ramp insensitive least square loss function with $\varepsilon=0.5$ and $\theta=1.5$. (f) Bounded least square loss function with $t=1$ and $\theta=2$.
• Figure 2: The proposed HawkEye loss function and its gradient for different values of $a$ and $\lambda$. Subfigures (a), (c) are plotted for fixed $\lambda$ and different values of $a$ while (b), (d) are plotted for fixed $a$ and different values of $\lambda$.
• Figure 3: Visual comparison of HawkEye loss and baseline loss functions. Subfigures (a), (b), (c), and (d) demonstrate the comparison of HawkEye loss function with the least square loss, insensitive loss, canal loss, and bounded least square loss, respectively.
• Figure 4: Flowchart illustrating the key stages of the HE-LSSVR model, showcasing the transformation from original input data to the decision function, highlighting the strategic integration of the HawkEye loss function and the application of representer theorem and Adam algorithm.

Theorems & Definitions (6)

• proof
• proof
• proof
• proof
• proof
• proof