Tilting the Odds at the Lottery: the Interplay of Overparameterisation and Curricula in Neural Networks

TL;DR

The work asks why curriculum learning often yields limited benefits in deep networks and shows that excessive overparameterisation can erase curriculum gains. By analyzing online learning on a XOR-like Gaussian Mixture (XGM) with a $2$-layer network in the mean-field limit ($d\to\infty$), it derives ODEs for order parameters that govern learning dynamics. The authors disentangle curriculum effects into two sub-tasks—discovering the relevant manifold and identifying the labeling rule—and demonstrate that increasing $K$ raises the likelihood of favorable initial lottery tickets, while diminishing curriculum advantages in the highly overparameterised regime. They validate predictions with experiments on real data (e.g., corrupted MNIST/CIFAR-10), showing the phenomenon generalises beyond the analytical model. Overall, the paper clarifies when curricula can help and when they are unlikely to, offering guidance on curriculum design in systems with varying degrees of overparameterisation.

Abstract

A wide range of empirical and theoretical works have shown that overparameterisation can amplify the performance of neural networks. According to the lottery ticket hypothesis, overparameterised networks have an increased chance of containing a sub-network that is well-initialised to solve the task at hand. A more parsimonious approach, inspired by animal learning, consists in guiding the learner towards solving the task by curating the order of the examples, i.e. providing a curriculum. However, this learning strategy seems to be hardly beneficial in deep learning applications. In this work, we undertake an analytical study that connects curriculum learning and overparameterisation. In particular, we investigate their interplay in the online learning setting for a 2-layer network in the XOR-like Gaussian Mixture problem. Our results show that a high degree of overparameterisation -- while simplifying the problem -- can limit the benefit from curricula, providing a theoretical account of the ineffectiveness of curricula in deep learning.

Figures (10)

• Figure 1: XOR-like Gaussian mixture and the lottery ticket.Panels (a) and (b) show the relevant coordinates in the input space of the Gaussian mixture considered in our theoretical framework. Circles and crosses represent the position of the centroids for the two classes. All the remaining dimensions are spurious, and their centroids are centered at zero. Panel (a) represents a possible 'bad' initialisation of a 2-layer neural network with $K=4$ hidden units (weight vectors depicted in gray), where 'bad' means that this will not lead to the optimal solution. Notice that $K=4$ is however the minimal number of parameters to optimally solve the task. Panel (b) considers instead an overparameterised neural network with $K=10$: notice that by increasing $K$ the probability of covering all 4 quadrants at initialisation increases, making it more likely to have a good 'lottery ticket'. Panel (c) shows that, indeed, overparameterisation consistently helps achieve a lower generalisation error (y-axis), even as noise intensity $\sigma$ increases. Eventually, the clusters become so overlapping that even an overparameterised system cannot achieve good performance.
• Figure 2: The benefits of a curriculum.Panel (a) shows that curriculum learning ('c') can achieve a better test loss in comparison with random ordering ('rand') and no-fading ('nf') paradigms. Consistent with results in cognitive science, neural networks find benefits from curricula in a Goldilocks range of the noise. Histograms show centroid coverage for the three protocols. Panel (b) highlights another aspect of curricula that comes from the dynamics: curricula can also speed up learning considerably.
• Figure 3: Scrutinising the curriculum gain.Panel (a) shows a controlled experiment where a $K=4$-network is initialised with 3 neurons in the optimal configuration, and the remaining one at an angle $\theta_1$ with the free centroid and with a fraction $\rho_1=0.1$ of norm lying on the 2-dimensional relevant manifold. The experiment is run in the high-noise regime, $\sigma=1.0$. The curriculum (blue) and random (red) lines show that curriculum finds the optimal configuration for a significantly larger range of initial angles. Furthermore, the inset shows that the final $\rho_1$ of the free neuron is consistently larger for curriculum. Panel (b) shows the instantaneous benefit of curricula, both for the rate of alignment with the left-out centroid---$d\,M_{11}$ (upper row)---and in the rate of alignment with the relevant manifold---$d\,\rho_1$ (lower row). The heatmaps span the fading factors $\mu$ on the horizontal axis, and the current mass on the relevant manifold $\rho_1$ (alignment with the centroid $m_1$) on the vertical axis of the upper (lower) row. Crucially, at a given $\mu$, the benefit is non-monotonic on the other variable, implying that there is an optimal period during learning for using curricula effectively.
• Figure 4: Interaction between curricula and overparameterisation.Panel (a) shows histograms of the cluster coverage achieved after 10,000 epochs, for a standard deviation $\sigma=0.4$ and 1,000 easy samples ($\alpha=0.1$). The colours (blue, red, and yellow) denote the strategies used (curriculum, random order, and no-fading, respectively) during training. The different plots in the panel show the effect of overparameterisation, controlled by the parameter $K$. While a small degree of overparameterisation benefits curricula more than other strategies, a very large overparameterisation makes all strategies equally effective. Panel (b) generalises the picture to a broad range of $\sigma$s and larger training time. In all these cases, we always keep the number of 'easy' samples fixed to 1,000---e.g. in the rightmost plot only 0.1% of the samples are easy. The different lines represent the gap in noiseless generalisation error between curriculum and no-fading (solid lines), and random order and no-fading (dotted lines), for different network parameterisations $K$.
• Figure 5: Interplay between curriculum and overparameterisation in real data.Panel (a) shows the effect of curriculum and overparameterisation for a MLP trained in a corrupted fashionMNIST dataset. We can observe two types of overparameterisation, layer-wise (x-axis) and in depth (bars), and their impact on the gap between curriculum and random order (y-axis). The key observation is that, above the Goldilocks range, both kinds of overparameterisation reduce the benefit of curricula. Panel (b) extends the analysis on CNNs trained in a corrupted CIFAR10 dataset. As we overparameterise by increasing the number of filters (x-axis), the accuracies (y-axis) of the two strategies get closer and their gap decreases. In Panel (c) we show some of the samples used in training and testing. For the MLP, we add 'difficulty' to fashionMNIST by adding white noise to the images. Since CNNs are robust against this kind of perturbation, we increase the 'difficulty' of CIFAR10 samples by adding a distracting frame around the image. The different columns represent the easy samples, the test samples and the hard samples, respectively.