Support Vector Regression: Risk Quadrangle Framework

TL;DR

This work embeds Support Vector Regression (SVR) into the Risk Quadrangle (RQ) framework, unifying SVR with risk management and distributionally robust optimization. It shows that ε-SVR minimizes the Vapnik error while ν-SVR minimizes the CVaR norm, making SVR an estimator of the average of two symmetric conditional quantiles. The authors prove the equivalence of ε-SVR and ν-SVR in general stochastic settings, reinterpret ν-SVR as Distributionally Robust Regression (DRR), and derive a new dual formulation that enables kernelization with improved computational efficiency. A case study corroborates the theoretical results, and the work provides a practical pathway to leverage SVR for risk-sensitive and robust regression tasks.

Abstract

This paper investigates Support Vector Regression (SVR) within the framework of the Risk Quadrangle (RQ) theory. Every RQ includes four stochastic functionals -- error, regret, risk, and \emph{deviation}, bound together by a so-called statistic. The RQ framework unifies stochastic optimization, risk management, and statistical estimation. Within this framework, both $\varepsilon$-SVR and $ν$-SVR are shown to reduce to the minimization of the \emph{Vapnik error} and the Conditional Value-at-Risk (CVaR) norm, respectively. The Vapnik error and CVaR norm define quadrangles with a statistic equal to the average of two symmetric quantiles. Therefore, RQ theory implies that $\varepsilon$-SVR and $ν$-SVR are asymptotically unbiased estimators of the average of two symmetric conditional quantiles. Moreover, the equivalence between $\varepsilon$-SVR and $ν$-SVR is demonstrated in a general stochastic setting. Additionally, SVR is formulated as a deviation minimization problem. Another implication of the RQ theory is the formulation of $ν$-SVR as a Distributionally Robust Regression (DRR) problem. Finally, an alternative dual formulation of SVR within the RQ framework is derived. Theoretical results are validated with a case study.

Figures (1)

• Figure 1: Graphical illustration of the SVR estimator. For $\varepsilon$-SVR, let $\mathbb{E}[|Z_f| - \varepsilon]_+$ be the Vapnik error with parameter $\varepsilon \geq 0$, where $Z_f$ denotes the regression residual with pdf $f_{Z_f}(z)$. The parameter $\varepsilon$ defines the distance between two symmetric quantiles, $q_{\frac{1+\alpha}{2}}(Z_f)$ and $q_{\frac{1-\alpha}{2}}(Z_f)$, such that $\varepsilon = \frac{1}{2}(q_{\frac{1+\alpha}{2}}(Z_f) - q_{\frac{1-\alpha}{2}}(Z_f))$. The optimal solution to the regression problem with the Vapnik error is then the average of these two conditional quantiles, where the parameter $\alpha$ is implicitly defined by $\varepsilon$. For $\nu$-SVR with $\nu = 1 - \alpha$, one specifies the parameter $\alpha$ directly and minimizes the CVaR norm.

Theorems & Definitions (42)

• Definition 3.1: Regular Risk Measure, Quadrangle
• Definition 3.2: Quantile
• Remark 3.1: Sum and scaling of quantiles
• Definition 3.3: CVaR
• Theorem 3.1: CVaR Optimization Formula
• Theorem 3.2: Dual CVaR Optimization Formula
• Remark 3.2
• Definition 3.4: Regular Error Measure, Quadrangle
• Definition 3.5: Scaled CVaR Norm
• Definition 3.6: Non-scaled CVaR Norm