Totally Concave Regression

TL;DR

This work introduces totally concave regression, a multivariate, shape-constrained regression framework that extends univariate concave regression via total concavity (Popoviciu). It formulates a least-squares estimator over a convex function class with measure-based representations, enables a finite-dimensional NNLS computation, and provides strong risk and adaptivity guarantees under fixed and random designs. Regularization is proposed to curb boundary overfitting, and variants for high-dimensional settings balance flexibility and tractability. Empirical results on earnings, housing, and 401(k) data demonstrate competitive predictive performance and practical viability, with an accompanying R package for implementation.

Abstract

Shape constraints in nonparametric regression provide a powerful framework for estimating regression functions under realistic assumptions without tuning parameters. However, most existing methods$\unicode{x2013}$except additive models$\unicode{x2013}$impose too weak restrictions, often leading to overfitting in high dimensions. Conversely, additive models can be too rigid, failing to capture covariate interactions. This paper introduces a novel multivariate shape-constrained regression approach based on total concavity, originally studied by T. Popoviciu. Our method allows interactions while mitigating the curse of dimensionality, with convergence rates that depend only logarithmically on the number of covariates. We characterize and compute the least squares estimator over totally concave functions, derive theoretical guarantees, and demonstrate its practical effectiveness through empirical studies on real-world datasets.

Figures (8)

• Figure 1: Four types of multivariate concavity.
• Figure 2: Best performing and bottom two ratios of each method for various sub-sample sizes. The $x$-axis and $y$-axis represent the proportion of training data and the proportion of repetitions (out of 100) where (A) each model outperforms all the other four models and (B) each model is one of the two worst-performing models.
• Figure 3: Empirical cumulative distributions of model ranks across 100 splits into training and test sets. Ranks are determined by mean squared errors on the test sets.
• Figure 4: Fitted regression functions of (A) Ours 2 and (B) Int 3 for a random split where Ours 2 performs the worst. The regression functions are plotted against crime and room, with $\log(\textsf{nox})$, $\log(\textsf{distance})$, and stratio fixed at their median.
• Figure 5: Empirical cumulative model rank distributions across 100 training-test splits. Add. (Reg.) and Ours 2 (Reg.) are the regularized variants \ref{['modified-estimator']} of Additive and Ours 2.

Theorems & Definitions (58)

• Definition 1: Total concavity in the sense of Popoviciu
• Proposition 1
• Proposition 2
• Proposition 3
• Proposition 4
• Proposition 5
• Theorem 1
• Theorem 2
• Theorem 3
• Theorem 4