Early learning of the optimal constant solution in neural networks and humans

TL;DR

The OCS is suggested as a universal learning principle in supervised, error-corrective learning, and the mechanistic reasons for its prevalence are suggested.

Abstract

Deep neural networks learn increasingly complex functions over the course of training. Here, we show both empirically and theoretically that learning of the target function is preceded by an early phase in which networks learn the optimal constant solution (OCS) - that is, initial model responses mirror the distribution of target labels, while entirely ignoring information provided in the input. Using a hierarchical category learning task, we derive exact solutions for learning dynamics in deep linear networks trained with bias terms. Even when initialized to zero, this simple architectural feature induces substantial changes in early dynamics. We identify hallmarks of this early OCS phase and illustrate how these signatures are observed in deep linear networks and larger, more complex (and nonlinear) convolutional neural networks solving a hierarchical learning task based on MNIST and CIFAR10. We explain these observations by proving that deep linear networks necessarily learn the OCS during early learning. To further probe the generality of our results, we train human learners over the course of three days on the category learning task. We then identify qualitative signatures of this early OCS phase in terms of the dynamics of true negative (correct-rejection) rates. Surprisingly, we find the same early reliance on the OCS in the behaviour of human learners. Finally, we show that learning of the OCS can emerge even in the absence of bias terms and is equivalently driven by generic correlations in the input data. Overall, our work suggests the OCS as a universal learning principle in supervised, error-corrective learning, and the mechanistic reasons for its prevalence.

Figures (16)

• Figure 1: Universal early learning of the optimal constant solution (OCS). A Graphical illustration of our hypothesis. B The hierarchical learning task used across learners and the OCS solution $\hat{\mathbf{y}}_{ocs}$. C Learning task in linear networks with bias terms. D The task in CNNs. E The task for humans.
• Figure 2: Exact learning dynamics. A Deep linear networks with (left) and without (right) bias. B Shallow linear networks with (left) and without (right) bias. Top row: Exact and simulated $\mathbf{A}(t) \text{ and }\mathbf{B}(t)$ for deep and shallow networks respectively. Bottom row: Exact and simulated loss.
• Figure 3: Early learning is driven to the OCS. Top row: Network outputs for a single output unit in response to all inputs $\mathbf{x}_i$. Bottom row: Continuous correct-rejection scores $f^{tn}$ for the three hierarchical levels as indicated by colours (\ref{['app:continous-TPR-and-TNR-Rates']}) Performance approaches levels expected under the OCS (dotted lines). Left: Linear networks with bias terms. Center: CNNs. Right: Linear networks without bias terms.
• Figure 4: Loss curves for different bias variations.
• Figure 5: Early response bias towards the OCS across learners in the hierarchical learning task. Dashed red line indicates chance performance. Dashed vertical grey lines indicates breaks between days for human learners. Colour code as in \ref{['fig:outputs']}.

Theorems & Definitions (16)

• Proposition 1: Feasibility of closed-form learning dynamics
• Proposition 2: The OCS is linked to shared properties
• Theorem 1: Early learning is biased by the OCS mode
• Proposition 3: NTK of linear networks with bias terms
• Corollary 1: Input correlations induce early OCS
• Proposition 3: Feasibility of closed-form learning dynamics
• proof
• Proposition 3: The OCS is linked to shared properties
• proof
• Proposition 4: Continuous symmetry induces $\1$