Singular leaning coefficients and efficiency in learning theory
Miki Aoyagi
TL;DR
This work addresses learning efficiency in singular learning models, notably deep networks, by computing learning coefficients through the lens of log canonical thresholds and resolution of singularities. They derive explicit expressions for the coefficients $\lambda(w_0)$ and their order $\theta(w_0)$ for deep linear networks and three-layer networks with linear units, and then extend the analysis to ReLU networks, showing that ReLU coefficients coincide with linear ones via a region-based decomposition. The Softmax case is treated by reformulating the squared-output discrepancy in terms of logits differences to fit the same theoretical framework. These results underpin model-selection criteria such as WBIC and sBIC and provide theoretical insight into generalization, stability, and the behavior of deep learning models beyond regular settings, with potential practical impact on understanding double descent and learning dynamics.
Abstract
Singular learning models with non-positive Fisher information matrices include neural networks, reduced-rank regression, Boltzmann machines, normal mixture models, and others. These models have been widely used in the development of learning machines. However, theoretical analysis is still in its early stages. In this paper, we examine learning coefficients, which indicate the general learning efficiency of deep linear learning models and three-layer neural network models with ReLU units. Finally, we extend the results to include the case of the Softmax function.