Universal Inference Meets Random Projections: A Scalable Test for Log-concavity

TL;DR

The paper addresses the challenge of validly testing log-concavity of multivariate densities, introducing a universal likelihood ratio test (LRT) that achieves finite-sample validity under iid sampling. To scale to higher dimensions, it leverages dimension reduction via axis-aligned and random projections, enabling one-dimensional log-concave MLEs and powerful, scalable testing. Empirical results show universal tests with dimension reduction often outperform permutation-based approaches, especially in higher dimensions, while maintaining validity; the full-dimensional approach can suffer from power loss as dimensionality grows. Theoretical contributions include a consistency result for the universal log-concavity test under mild regularity and estimability conditions, with detailed proofs and supportive simulations, highlighting the practical impact for shape-constrained density modeling in economics, reliability, and survival analysis.

Abstract

Shape constraints yield flexible middle grounds between fully nonparametric and fully parametric approaches to modeling distributions of data. The specific assumption of log-concavity is motivated by applications across economics, survival modeling, and reliability theory. However, there do not currently exist valid tests for whether the underlying density of given data is log-concave. The recent universal inference methodology provides a valid test. The universal test relies on maximum likelihood estimation (MLE), and efficient methods already exist for finding the log-concave MLE. This yields the first test of log-concavity that is provably valid in finite samples in any dimension, for which we also establish asymptotic consistency results. Empirically, we find that a random projections approach that converts the d-dimensional testing problem into many one-dimensional problems can yield high power, leading to a simple procedure that is statistically and computationally efficient.

Figures (19)

• Figure 1: Rejection proportions for tests of $H_0: f^*$ is log-concave versus $H_1: f^*$ is not log-concave. The permutation test from cule2010 is valid or approximately valid for $d\leq 3$, but it is not valid for $d\geq 4$. Our test that combines random projections and universal inference (Algorithm \ref{['alg:randproj']}) is provably valid for all $n$ and $d$ while having high power.
• Figure 2: Densities from fitting log-concave MLE on $n = 5000$ observations. The true density is the Normal mixture $f^*(x) = 0.5\phi_1(x) + 0.5\phi_1(x-\mu)$. In all settings, the LogConcDEAD and logcondens packages return similar results. In the $\|\mu\| = 0$ and $\|\mu\| = 2$ log-concave settings, the log-concave MLE is close to the true density. In the $\|\mu\| = 4$ non-log-concave setting, the log-concave densities appear to have normal tails and uniform centers.
• Figure 3: Rejection proportions for tests of $H_0: f^*$ is log-concave versus $H_1: f^*$ is not log-concave. When $d=1$, the permutation test and the full oracle universal test have similar power. The full oracle universal approach remains valid in higher dimensions, but it has low power for moderate $\|\mu\|$ when $d\geq 3$.
• Figure 4: Power of tests of $H_0: f^*$ is log-concave versus $H_1: f^*$ is not log-concave. $\mu$ vector for second component is $\mu = -(\|\mu\|, 0, \ldots, 0)$.
• Figure 5: Power of tests of $H_0: f^*$ is log-concave versus $H_1: f^*$ is not log-concave. $\mu$ vector for second component is $\mu = -(\|\mu\|d^{-1/2}, \|\mu\|d^{-1/2}, \ldots, \|\mu\|d^{-1/2})$.

Theorems & Definitions (23)

• Theorem 1: wasserman2020universal
• Theorem 2: Proposition 1(a) of cule2010
• Theorem 3
• Theorem 3: wasserman2020universal
• proof
• Lemma 1
• proof
• Theorem 4
• proof
• Lemma 2