Deep Quantile Regression: Mitigating the Curse of Dimensionality Through Composition

TL;DR

This work addresses nonparametric quantile regression when the conditional quantile function is a composition of low-dimensional components. It introduces Deep Quantile Regression (DQR), training deep ReLU networks via empirical risk minimization using the check loss $\rho_\tau$, and establishes non-asymptotic excess risk and MISE bounds that depend on the intrinsic dimensions of the composition rather than the ambient dimension. A key technical contribution is new approximation results for composite functions by neural networks, showing the error rate depends on component dimensions with prefactors that grow only polynomially in the ambient dimension, thereby mitigating the curse of dimensionality under compositional structure. The paper also demonstrates, through extensive numerical studies, that DQR outperforms kernel-based nonparametric QR in settings with nonlinear or high-dimensional structure, suggesting practical advantages for high-dimensional quantile estimation in statistics and econometrics.

Abstract

This paper considers the problem of nonparametric quantile regression under the assumption that the target conditional quantile function is a composition of a sequence of low-dimensional functions. We study the nonparametric quantile regression estimator using deep neural networks to approximate the target conditional quantile function. For convenience, we shall refer to such an estimator as a deep quantile regression (DQR) estimator. We show that the DQR estimator achieves the nonparametric optimal convergence rate up to a logarithmic factor determined by the intrinsic dimension of the underlying compositional structure of the conditional quantile function, not the ambient dimension of the predictor. Therefore, DQR is able to mitigate the curse of dimensionality under the assumption that the conditional quantile function has a compositional structure. To establish these results, we analyze the approximation error of a composite function by neural networks and show that the error rate only depends on the dimensions of the component functions. We apply our general results to several important statistical models often used in mitigating the curse of dimensionality, including the single index, the additive, the projection pursuit, the univariate composite, and the generalized hierarchical interaction models. We explicitly describe the prefactors in the error bounds in terms of the dimensionality of the data and show that the prefactors depends on the dimensionality linearly or quadratically in these models. We also conduct extensive numerical experiments to evaluate the effectiveness of DQR and demonstrate that it outperforms a kernel-based method for nonparametric quantile regression.

Figures (12)

• Figure 1: The fitted quantile curves by different methods under the univariate model "Wave" with "Sine" error. The training data is depicted as grey dots.The target quantile functions at the quantile levels $\tau=$0.25 (yellow), 0.5 (green), 0.75 (red) are depicted as solid curves, and the estimated quantile functions are represented by dashed curves with the same color. From the left to right, the subfigures correspond to the methods: DQR, kernel QR and linear QR. The fitted DLS curve (in blue) is included in the left subfigure.
• Figure 2: The target quantiles curves at $\tau=0.25, 0.50. 0.75$ under different models and error distributions. From the left to the right, each column corresponds a data generation model, "Linear", "Wave" and "Triangle". From the top to the bottom, each row corresponds a error distribution, $t(3)$, "Sine" and "Exp".
• Figure 3: The fitted quantile curves by different methods under the univariate model "Linear" with different errors. The training data is depicted as grey dots.The target quantile functions at the quantile levels $\tau=$0.25 (yellow), 0.5 (green), 0.75 (red) are depicted as solid curves, and the estimated quantile functions are represented by dashed curves with the same color. From the top to the bottom, the rows correspond to the errors $t(3)$, "Sine" and "Exp". From the left to the right, the subfigures correspond to the methods DQR, kernel QR and linear QR. The fitted DLS curve (in blue) is included in the left subfigure.
• Figure 4: The fitted quantile curves by different methods under the univariate model "Wave" with different errors. The training data is depicted as grey dots.The target quantile functions at the quantile levels $\tau=$0.25 (yellow), 0.5 (green), 0.75 (red) are depicted as solid curves, and the estimated quantile functions are represented by dashed curves with the same color. From the top to the bottom, the rows correspond to the errors $t(3)$, "Sine" and "Exp". From the left to the right, the subfigures correspond to the methods DQR, kernel QR and linear QR. The fitted DLS curve (in blue) is included in the left subfigure.
• Figure 5: The fitted quantile curves by different methods under the univariate model "Triangle" with different errors. The training data is depicted as grey dots.The target quantile functions at the quantile levels $\tau=$0.25 (yellow), 0.5 (green), 0.75 (red) are depicted as solid curves, and the estimated quantile functions are represented by dashed curves with the same color. From the top to the bottom, the rows correspond to the errors $t(3)$, "Sine" and "Exp". From the left to the right, the subfigures correspond to the methods DQR, kernel QR and linear QR. The fitted DLS curve (in blue) is included in the left subfigure.

Theorems & Definitions (36)

• Lemma 1
• Lemma 2
• Remark 1
• Lemma 3
• Remark 2
• Lemma 4
• Remark 3
• Theorem 1: Non-asymptotic excess risk bound
• Remark 4
• Corollary 1