Semi-parametric bulk and tail regression using spline-based neural networks

TL;DR

This work combines SPQR with the blended generalised Pareto distribution to create semi-parametric quantile regression for extremes (SPQRx), which provides a flexible semi-parametric approach to density regression that is compliant with traditional EVT.

Abstract

Semi-parametric quantile regression (SPQR) is a flexible approach to density regression that learns a spline-based representation of conditional density functions using neural networks. As it makes no parametric assumptions about the underlying density, SPQR performs well for in-sample testing and interpolation. However, it can perform poorly when modelling heavy-tailed data or when asked to extrapolate beyond the range of observations, as it fails to satisfy any of the asymptotic guarantees provided by extreme value theory (EVT). To build semi-parametric density regression models that can be used for reliable tail extrapolation, we create the blended generalised Pareto (GP) distribution, which i) provides a model for the entire range of data and, via a smooth and continuous transition, ii) benefits from exact GP upper-tails without the need for intermediate threshold selection. We combine SPQR with our blended GP to create semi-parametric quantile regression for extremes (SPQRx), which provides a flexible semi-parametric approach to density regression that is compliant with traditional EVT. We handle interpretability of SPQRx through the use of model-agnostic variable importance scores, which provide the relative importance of a covariate for separately determining the bulk and tail of the conditional density. The efficacy of SPQRx is illustrated on simulated data, and an application to U.S. wildfire burnt areas from 1990-2020.

Figures (13)

• Figure 1: Illustrative examples of the blended GP distribution (left) and density (right) functions, with the constituent distribution modelled via SPQR. For a fixed random seed, we generate $K$ basis coefficients and construct the SPQR density and distribution functions (green curves). We then find $a=Q_{\rm SPQR}(p_a | \mathcal{W})$ and $b=Q_{\rm SPQR}(p_b|\mathcal{W})$, where $p_a=0.25$ and $p_b=0.9$. The values of $a$ and $b$ are denoted by the orange horizontal lines. Then, we find the required GP distribution function (red) to satisfy continuity of the bGP($\mathcal{W},\xi)$ distribution function; its resulting distribution and density functions are provided in black. The weighting function shape parameter $c_2$ is fixed to $c_2=5$ across all panels.
• Figure 2: Schematic of the underlying multi-layer percepton (MLP) of an SPQRx model with $H=2$ hidden layers. The input covariates are $\mathbf{x} =(x_1,\dots,x_p)$, and the output of the final $\hbox{softmax}^*(\cdot)$ layer is $(\xi(\mathbf{x}),w_1(\mathbf{x}),\dots,w_K(\mathbf{x})),$ where $w_1(\mathbf{x}),\dots,w_K(\mathbf{x})$ are the $K$ basis functions comprising the constituent SPQR density/distribution.
• Figure 3: Simulation study: estimates of density (top) and log-survival (bottom) functions for three test covariate vectors. The red dashed lines give the true functions. The black and green curves are the corresponding estimates from SPQR and SPQRx, respectively. The values of $a$ and $b$ (from the SPQRx fit) are denoted by the orange horizontal lines.
• Figure 4: Simulation study: estimates of the variable importance scores $\hbox{VI}_j\left(\hat{Q}(\tau|\mathbf{x})\right)$ for the covariates $X_j,j=1,2,3,$ as a function of the quantile level $\tau$.
• Figure 5: Goodness-of-fit diagnostics for the SPQR and SPQRx models. Pooled PP (top) and QQ (bottom) plots for the fitted SPQR (left) and SPQRx (right) models, evaluated on the test data. The red points in the right column denote the two most extreme test values, which cannot be modelled using SPQR and thus do not appear on the left column. The grey area bounded by the dashed lines represent the $95\%$ bootstrap confidence intervals.