The Hellinger Bounds on the Kullback-Leibler Divergence and the Bernstein Norm
Tetsuya Kaji
TL;DR
This work addresses when the Hellinger distance can upper-bound key discrepancy measures—Kullback--Leibler divergence, Kullback--Leibler variation, and the Bernstein norm—within likelihood-based nonparametric models, including settings with unbounded likelihood ratios. It derives necessary and sufficient conditions for each bound in terms of tail behavior on the unfavorable region where $p_0/p>4$, unifying the Bernstein, KL, and moment-type criteria under a common Hellinger framework. The main contributions are sharp, δ-dependent bounds for the fractional Bernstein norm of log-likelihood ratios and for KL-type divergences, plus a thorough comparison to existing conditions, establishing when these bounds hold and how they relate to uniform boundedness and local moment assumptions. The results enable relaxing regularity conditions in sieve maximum likelihood estimation, yielding convergence rates under weaker assumptions and with unbounded likelihood ratios, and they offer insights applicable to posterior contraction analyses in nonparametric Bayesian contexts.
Abstract
The Kullback-Leibler divergence, the Kullback-Leibler variation, and the Bernstein "norm" are used to quantify discrepancies among probability distributions in likelihood models such as nonparametric maximum likelihood and nonparametric Bayes. They are closely related to the Hellinger distance, which is often easier to work with. Consequently, it is of interest to characterize conditions under which the Hellinger distance serves as an upper bound for these measures. This article characterizes a necessary and sufficient condition for each of the discrepancy measures to be bounded by the Hellinger distance. It accommodates unbounded likelihood ratios and generalizes all previously known results. We then apply it to relax the regularity condition for the sieve maximum likelihood estimator.