On the sample complexity of parameter estimation in logistic regression with normal design
Daniel Hsu, Arya Mazumdar
TL;DR
This work characterizes the non-asymptotic sample complexity for estimating the logistic regression parameter under a normal design, revealing a three-regime dependence on the inverse temperature $\beta$. The authors establish matching lower and upper bounds (up to polylog factors) across high, moderate, and low temperature regimes, using Fano’s inequality for lower bounds and regime-specific estimators (linear, ReLU-based ERM, and ERM) for upper bounds. The results connect to noisy one-bit sensing and PAC learning, and they quantify how the difficulty of parameter estimation transitions from $\Theta(d/\epsilon^2)$-like behavior at high temperatures to $\Theta(d/\epsilon)$ in the zero-temperature limit, with intermediate scalings $d/(\beta^2\epsilon^2)$ and $d/(\beta\epsilon^2)$. These insights clarify when estimation becomes fundamentally easier or harder as noise levels change, and they prompt further work on computation-friendly estimators that attain the optimal rates across all regimes.
Abstract
The logistic regression model is one of the most popular data generation model in noisy binary classification problems. In this work, we study the sample complexity of estimating the parameters of the logistic regression model up to a given $\ell_2$ error, in terms of the dimension and the inverse temperature, with standard normal covariates. The inverse temperature controls the signal-to-noise ratio of the data generation process. While both generalization bounds and asymptotic performance of the maximum-likelihood estimator for logistic regression are well-studied, the non-asymptotic sample complexity that shows the dependence on error and the inverse temperature for parameter estimation is absent from previous analyses. We show that the sample complexity curve has two change-points in terms of the inverse temperature, clearly separating the low, moderate, and high temperature regimes.