Nonconvex Penalized LAD Estimation in Partial Linear Models with DNNs: Asymptotic Analysis and Proximal Algorithms
Lechen Feng, Haoran Li, Lucky Li, Xingqiu Zhao
TL;DR
This work develops a robust, high-dimensional framework for estimating a partially linear model with a nonparametric term $g_0$ represented by sparse Deep Neural Networks under Least Absolute Deviation loss. It establishes consistency, convergence rates $O_p(r_N\log^2 N+\lambda_N)$ for both the parametric and nonparametric components, and asymptotic normality for the parametric part, leveraging infinite-dimensional variational analysis and entropy control to handle nonconvex, nonsmooth regularization and growing networks. The paper also analyzes the oracle problem and two relaxation-based optimization paths: a continuous-approximation proximal scheme with cheaper updates and a non-approximation approach maintaining exact sparsity, each with a rigorous convergence framework and a Weak Sard property. Collectively, the results provide a theoretical foundation for robust, scalable semi-parametric inference in PLMs with flexible, DNN-based nonparametric terms and offer practical guidance on algorithmic choices balancing statistical accuracy and computation.
Abstract
This paper investigates the partial linear model by Least Absolute Deviation (LAD) regression. We parameterize the nonparametric term using Deep Neural Networks (DNNs) and formulate a penalized LAD problem for estimation. Specifically, our model exhibits the following challenges. First, the regularization term can be nonconvex and nonsmooth, necessitating the introduction of infinite dimensional variational analysis and nonsmooth analysis into the asymptotic normality discussion. Second, our network must expand (in width, sparsity level and depth) as more samples are observed, thereby introducing additional difficulties for theoretical analysis. Third, the oracle of the proposed estimator is itself defined through a ultra high-dimensional, nonconvex, and discontinuous optimization problem, which already entails substantial computational and theoretical challenges. Under such the challenges, we establish the consistency, convergence rate, and asymptotic normality of the estimator. Furthermore, we analyze the oracle problem itself and its continuous relaxation. We study the convergence of a proximal subgradient method for both formulations, highlighting their structural differences lead to distinct computational subproblems along the iterations. In particular, the relaxed formulation admits significantly cheaper proximal updates, reflecting an inherent trade-off between statistical accuracy and computational tractability.