Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Finite-sample performance of the maximum likelihood estimator in logistic regression

Hugo Chardon, Matthieu Lerasle, Jaouad Mourtada

TL;DR

The paper develops sharp finite-sample guarantees for the maximum likelihood estimator in logistic regression, focusing on both the existence of the MLE and its predictive excess logistic risk. It first establishes optimal, non-asymptotic thresholds for the Gaussian-design, well-specified setting and then extends to regular-design settings under a two-dimensional margin condition, as well as to misspecified models. The results cover both well-specified and misspecified cases, with detailed gradient and Hessian analyses that underpin a convex localization approach. The findings illuminate phase-transition-like behavior and the role of design structure (Gaussian, regular, Bernoulli-like) in determining the MLE’s performance, providing practically interpretable sample-size requirements and risk bounds in high-dimensional logistic regression.

Abstract

Logistic regression is a classical model for describing the probabilistic dependence of binary responses to multivariate covariates. We consider the predictive performance of the maximum likelihood estimator (MLE) for logistic regression, assessed in terms of logistic risk. We consider two questions: first, that of the existence of the MLE (which occurs when the dataset is not linearly separated), and second that of its accuracy when it exists. These properties depend on both the dimension of covariates and on the signal strength. In the case of Gaussian covariates and a well-specified logistic model, we obtain sharp non-asymptotic guarantees for the existence and excess logistic risk of the MLE. We then generalize these results in two ways: first, to non-Gaussian covariates satisfying a certain two-dimensional margin condition, and second to the general case of statistical learning with a possibly misspecified logistic model. Finally, we consider the case of a Bernoulli design, where the behavior of the MLE is highly sensitive to the parameter direction.

Finite-sample performance of the maximum likelihood estimator in logistic regression

TL;DR

The paper develops sharp finite-sample guarantees for the maximum likelihood estimator in logistic regression, focusing on both the existence of the MLE and its predictive excess logistic risk. It first establishes optimal, non-asymptotic thresholds for the Gaussian-design, well-specified setting and then extends to regular-design settings under a two-dimensional margin condition, as well as to misspecified models. The results cover both well-specified and misspecified cases, with detailed gradient and Hessian analyses that underpin a convex localization approach. The findings illuminate phase-transition-like behavior and the role of design structure (Gaussian, regular, Bernoulli-like) in determining the MLE’s performance, providing practically interpretable sample-size requirements and risk bounds in high-dimensional logistic regression.

Abstract

Logistic regression is a classical model for describing the probabilistic dependence of binary responses to multivariate covariates. We consider the predictive performance of the maximum likelihood estimator (MLE) for logistic regression, assessed in terms of logistic risk. We consider two questions: first, that of the existence of the MLE (which occurs when the dataset is not linearly separated), and second that of its accuracy when it exists. These properties depend on both the dimension of covariates and on the signal strength. In the case of Gaussian covariates and a well-specified logistic model, we obtain sharp non-asymptotic guarantees for the existence and excess logistic risk of the MLE. We then generalize these results in two ways: first, to non-Gaussian covariates satisfying a certain two-dimensional margin condition, and second to the general case of statistical learning with a possibly misspecified logistic model. Finally, we consider the case of a Bernoulli design, where the behavior of the MLE is highly sensitive to the parameter direction.

Paper Structure

This paper contains 86 sections, 52 theorems, 490 equations.

Table of Contents

  1. Introduction
  2. Problem setting and main questions
  3. Existing results
  4. Classical asymptotics
  5. High-dimensional asymptotics.
  6. Non-asymptotic guarantees.
  7. Summary of contributions
  8. Gaussian design, well-specified model.
  9. Regular design, well-specified model.
  10. Regular design, misspecified model.
  11. Distributions satisfying the regularity assumptions.
  12. Additional related work
  13. Logistic regression as convex statistical learning.
  14. High-dimensional asymptotics.
  15. Worst-case design distributions, improper and robust estimators.
  16. ...and 71 more sections

Key Result

Theorem 1

Assume that the design $X \sim \mathsf{N} (0,I_{d})$ is Gaussian and that the model is well-specified with parameter $\theta^* \in \mathbb R^d$, and let $B=\max\{e,\|\theta^*\|\}$. There exist universal constants $c_1,c_2, c_3 > 0$ such that, for any $t > 0$: if then, with probability $1-e^{-t}$, the MLE $\widehat{\theta}_n$ exists and satisfies Moreover, for any $d\geqslant 53$ and $t \geqslant

Theorems & Definitions (101)

  • Theorem 1
  • Theorem 2
  • Remark 1
  • Proposition 1
  • Definition 1
  • Theorem 3
  • Theorem 4
  • Proposition 2
  • Proposition 3
  • proof
  • ...and 91 more