Online Learning of Neural Networks
Amit Daniely, Idan Mehalel, Elchanan Mossel
TL;DR
The paper analyzes online learning for feedforward neural networks with sign activation by connecting mistake bounds to the geometric TS-packing number, providing both upper and lower bounds in the general setting. It introduces a meta-online learner based on multiclass Weighted Majority and proves that the bound scales with $\mathtt{TS}(d,\gamma_1)/\gamma_1^2$, with sharp lower bounds achievable via TS-packings. To achieve dimension-free performance, the authors propose two natural restrictions: the multi-index model, reducing effective dimension to $k$, and an extended margin regime that yields bounds scaling with $\log Y$ and $\gamma$ across depth $L$, plus adaptive and agnostic extensions. The work also links to pruning methodologies by showing that a small subset of neurons suffices under margin-based pruning, and outlines open questions on TS-packing bounds and computational efficiency, highlighting both theoretical and practical directions for online neural-network learning.
Abstract
We study online learning of feedforward neural networks with the sign activation function that implement functions from the unit ball in $\mathbb{R}^d$ to a finite label set $\{1, \ldots, Y\}$. First, we characterize a margin condition that is sufficient and in some cases necessary for online learnability of a neural network: Every neuron in the first hidden layer classifies all instances with some margin $γ$ bounded away from zero. Quantitatively, we prove that for any net, the optimal mistake bound is at most approximately $\mathtt{TS}(d,γ)$, which is the $(d,γ)$-totally-separable-packing number, a more restricted variation of the standard $(d,γ)$-packing number. We complement this result by constructing a net on which any learner makes $\mathtt{TS}(d,γ)$ many mistakes. We also give a quantitative lower bound of approximately $\mathtt{TS}(d,γ) \geq \max\{1/(γ\sqrt{d})^d, d\}$ when $γ\geq 1/2$, implying that for some nets and input sequences every learner will err for $\exp(d)$ many times, and that a dimension-free mistake bound is almost always impossible. To remedy this inevitable dependence on $d$, it is natural to seek additional natural restrictions to be placed on the network, so that the dependence on $d$ is removed. We study two such restrictions. The first is the multi-index model, in which the function computed by the net depends only on $k \ll d$ orthonormal directions. We prove a mistake bound of approximately $(1.5/γ)^{k + 2}$ in this model. The second is the extended margin assumption. In this setting, we assume that all neurons (in all layers) in the network classify every ingoing input from previous layer with margin $γ$ bounded away from zero. In this model, we prove a mistake bound of approximately $(\log Y)/ γ^{O(L)}$, where L is the depth of the network.