Is model selection possible for the $\ell_p$-loss? PCO estimation for regression models

TL;DR

This work develops a model-selection framework for non-Euclidean losses in a sub-Gaussian sequence model by introducing the Penalized Comparison to Overfitting (PCO) procedure for weighted $\ell_p$ losses with $p\ge 1$. A general oracle inequality is established, with concentration results for sums of $|\xi_{\lambda}|^p$ via sub-Weibull bounds guiding penalty design; refined inequalities accommodate large model collections, revealing an elbow at $p=2$. The authors derive minimax convergence rates on Besov spaces for both constant and non-constant weights, identifying homogeneous, intermediate, and sparse regimes and proving adaptivity of the PCO estimator (including logarithmic losses in some cases). They extend the approach to nonparametric regression using wavelet representations, obtaining adaptive $\mathbb{L}_p$-risk rates for functions in Besov spaces and outlining the core steps of the functional-proof strategy. Overall, the paper offers a practically efficient single-minimization model-selection method that achieves near-optimal risk bounds for a broad class of Besov-sparse signals under non-Euclidean loss.

Abstract

This paper addresses the problem of model selection in the sequence model $Y=θ+\varepsilonξ$, when $ξ$ is sub-Gaussian, for non-euclidian loss-functions. In this model, the Penalized Comparison to Overfitting procedure is studied for the weighted $\ell_p$-loss, $p\geq 1.$ Several oracle inequalities are derived from concentration inequalities for sub-Weibull variables. Using judicious collections of models and penalty terms, minimax rates of convergence are stated for Besov bodies $\mathcal{B}_{r,\infty}^s$. These results are applied to the functional model of nonparametric regression.

Theorems & Definitions (32)

• Theorem 2.1
• Remark 2.2
• Theorem 2.3
• Corollary 2.4
• proof
• Theorem 2.5
• Remark 2.6
• Theorem 2.7
• Theorem 3.1
• Theorem 3.2