Overfitting has a limitation: a model-independent generalization gap bound based on Rényi entropy
Atsushi Suzuki, Jing Wang
TL;DR
This work introduces a model-independent upper bound on the generalization gap that depends solely on the Rényi entropy of the data-generating distribution, applied to symmetric (histogram-determined) learning algorithms. By leveraging the countable-data-space setting and the method of types, the authors connect generalization performance to distribution unevenness, not model size, and derive tight data-length requirements linked to exp(H_α(Q)). They extend no-free-lunch results to nonuniform distributions and show how randomizing labels (increasing Rényi entropy) degrades generalization in a quantified way. While providing broad theoretical insights, they acknowledge limitations in estimating Rényi entropy in practice and discuss future directions including stochastic symmetric algorithms and stronger assumptions. Overall, the paper offers a distribution-centered explanation for generalization behavior that complements existing complexity-based analyses and highlights when ultra-large models can generalize with sufficient data relative to entropy.
Abstract
Will further scaling up of machine learning models continue to bring success? A significant challenge in answering this question lies in understanding generalization gap, which is the impact of overfitting. Understanding generalization gap behavior of increasingly large-scale machine learning models remains a significant area of investigation, as conventional analyses often link error bounds to model complexity, failing to fully explain the success of extremely large architectures. This research introduces a novel perspective by establishing a model-independent upper bound for generalization gap applicable to algorithms whose outputs are determined solely by the data's histogram, such as empirical risk minimization or gradient-based methods. Crucially, this bound is shown to depend only on the Rényi entropy of the data-generating distribution, suggesting that a small generalization gap can be maintained even with arbitrarily large models, provided the data quantity is sufficient relative to this entropy. This framework offers a direct explanation for the phenomenon where generalization performance degrades significantly upon injecting random noise into data, where the performance degrade is attributed to the consequent increase in the data distribution's Rényi entropy. Furthermore, we adapt the no-free-lunch theorem to be data-distribution-dependent, demonstrating that an amount of data corresponding to the Rényi entropy is indeed essential for successful learning, thereby highlighting the tightness of our proposed generalization bound.