Precise Error Rates for Computationally Efficient Testing

TL;DR

This paper analyzes the fundamental limits of hypothesis testing under computational constraints in high-dimensional settings, focusing on the spiked Wigner model. It introduces a two-pronged approach: first, establishing the statistical optimum of linear spectral statistics (LSS) via a refined second-moment analysis and a positive achievability result for the ROC curve φ_λ, and second, deriving conditional hardness results for any polynomial-time test by leveraging an enhanced low-degree framework. Under a strengthened low-degree conjecture, the authors show that no subexponential-time test can surpass the ROC curve achieved by LSS for λ < 1, implying that the spectrum is a computationally universal statistic for testing in this regime. The work also provides an alternative route to known statistical limits and highlights a clear separation between what is statistically possible and what is efficiently computable, with implications for average-case complexity in high-dimensional testing. Overall, the paper offers a novel methodology to quantify precisely the testing error achievable under computational constraints and identifies the spectrum as a pivotal, computationally sufficient statistic in this setting.

Abstract

We revisit the fundamental question of simple-versus-simple hypothesis testing with an eye towards computational complexity, as the statistically optimal likelihood ratio test is often computationally intractable in high-dimensional settings. In the classical spiked Wigner model with a general i.i.d. spike prior we show (conditional on a conjecture) that an existing test based on linear spectral statistics achieves the best possible tradeoff curve between type I and type II error rates among all computationally efficient tests, even though there are exponential-time tests that do better. This result is conditional on an appropriate complexity-theoretic conjecture, namely a natural strengthening of the well-established low-degree conjecture. Our result shows that the spectrum is a sufficient statistic for computationally bounded tests (but not for all tests). To our knowledge, our approach gives the first tool for reasoning about the precise asymptotic testing error achievable with efficient computation. The main ingredients required for our hardness result are a sharp bound on the norm of the low-degree likelihood ratio along with (counterintuitively) a positive result on achievability of testing. This strategy appears to be new even in the setting of unbounded computation, in which case it gives an alternate way to analyze the fundamental statistical limits of testing.

Figures (1)

• Figure 1: Illustration of the proof of Theorem \ref{['thm:main-stat']}. Left: The curve $\phi = \phi_\lambda$ for $\lambda = 0.9$. All points $(\alpha,\beta)$ below the curve $\phi$ (and above the "trivial" dashed line) are achievable. Right: If hypothetically there were an achievable point $(\alpha^*,\beta^*)$ above the curve $\phi$ then every point below $\psi$ --- the upper concave envelope of $\phi$ and $(\alpha^*,\beta^*)$ --- would also be achievable. The improved curve $\psi$ would allow us to construct a function $f$ that makes the ratio in \ref{['eq:op-def-wig']} equal to $\mathsf{val}(\psi)$, which is strictly greater than $\mathsf{val}(\phi_\lambda) = (1-\lambda^2)^{-1/4}$, contradicting \ref{['eq:L-bound-rad']}.

Theorems & Definitions (53)

• Definition 1.1: Spiked Wigner testing problem
• Theorem 1.2: Main result, informal
• Theorem 1.3: Special case of fund-limits-wigner, informal
• Remark 2.1: Terminology
• Definition 2.2
• Remark 2.3: Model of computation
• Conjecture 2.4: Low-degree conjecture, informal
• Theorem 2.5: Special case of fund-limits-wigner
• Theorem 2.6: Special case of opt-subopt, Theorem 3.10
• Proposition 2.7: Special case of Proposition \ref{['prop:reduction']}