On the existence of the maximum likelihood estimate and convergence rate under gradient descent for multi-class logistic regression
Dwight Nwaigwe, Marek Rychlik
TL;DR
This work proves the existence of the maximum likelihood estimate for multiclass logistic regression under label smoothing without requiring data separability, by leveraging a shift-invariant, locally strongly convex loss on a subspace. It then derives a constructive convergence-rate bound for gradient descent by analyzing the Hessian’s spectrum, expressing the Hessian in a Kronecker-product form and bounding its eigenvalues. The results provide explicit, data-dependent bounds on the Hessian and condition number, both for the full-rank case $N=D$ and the overdetermined case $N>D$, enabling practical contraction-rate guarantees. The analysis hinges on an operator-theoretic framework, translating neural-network-like optimization questions into linear-algebraic bounds tied to data geometry and class-probability structure, with clear implications for convergence behavior in large-scale multiclass settings.
Abstract
We revisit the problem of the existence of the maximum likelihood estimate for multi-class logistic regression. We show that one method of ensuring its existence is by assigning positive probability to every class in the sample dataset. The notion of data separability is not needed, which is in contrast to the classical set up of multi-class logistic regression in which each data sample belongs to one class. We also provide a general and constructive estimate of the convergence rate to the maximum likelihood estimate when gradient descent is used as the optimizer. Our estimate involves bounding the condition number of the Hessian of the maximum likelihood function. The approaches used in this article rely on a simple operator-theoretic framework.