An efficient algorithm for kernel quantile regression
Shengxiang Deng, Xudong Li, Yangjing Zhang
TL;DR
This work tackles the scalability challenge of kernel quantile regression (KQR) by introducing a two-phase optimization framework that couples a fast inexact ADMM warm-start with a high-accuracy semismooth Newton augmented Lagrangian method (SSN-ALM). A Nyström-based preconditioner, built from a low-rank RKHS approximation via RPCholesky, dramatically improves the conditioning of the linear systems appearing in the Newton steps, enabling efficient preconditioned CG solves with near $O(n^2)$ cost for the inner solves. The method demonstrates strong empirical performance on synthetic and real energy-forecasting data, delivering substantial speedups over state-of-the-art solvers such as Gurobi and fastKQR while maintaining robustness and accuracy. These advances significantly extend the practical applicability of KQR to large-scale datasets and nonlinear conditional quantile modeling.
Abstract
Kernel Quantile Regression (KQR) extends classical quantile regression to nonlinear settings using kernel methods, offering powerful tools for modeling conditional distributions. However, its application to large-scale datasets is severely limited by the computational burden of the large, dense kernel matrix and the need to efficiently solve large, often ill-conditioned linear systems. Existing state-of-the-art solvers usually struggle with scalability. In this paper, we propose a novel and highly efficient two-phase optimization algorithm tailored for large-scale KQR. In the first phase, we employ an inexact alternating direction method of multipliers (ADMM) to compute a high-quality warm-start solution. The second phase refines this solution using an efficient semismooth Newton augmented Lagrangian method (ALM). A key innovation of our approach is a specialized preconditioning strategy that leverages low-rank approximations of the kernel matrix to effectively mitigate the ill-conditioning of the linear systems in the Newton steps of the ALM. This can significantly accelerate iterative solvers for linear systems. Extensive numerical experiments demonstrate that our algorithm substantially outperforms existing state-of-the-art commercial and specialized KQR solvers in terms of speed and scalability.