Divine Benevolence is an $x^2$: GLUs scale asymptotically faster than MLPs

TL;DR

The ``Gated Quadratic Unit'' is proposed, which has an even steeper $L(P)$ slope than the GLU and MLP, which opens the possibility of architecture design from first principles numerical theory to unlock superior scaling in large models.

Abstract

Scaling laws can be understood from ground-up numerical analysis, where traditional function approximation theory can explain shifts in model architecture choices. GLU variants now dominate frontier LLMs and similar outer-product architectures are prevalent in ranking models. The success of these architectures has mostly been left as an empirical discovery. In this paper, we apply the tools of numerical analysis to expose a key factor: these models have an $x^2$ which enables \emph{asymptotically} faster scaling than MLPs. GLUs have piecewise quadratic functional forms that are sufficient to exhibit quadratic order of approximation. Our key contribution is to demonstrate that the $L(P)$ scaling slope is $L(P)\propto P^{-3}$ for GLUs but only $L(P)=P^{-2}$ for MLPs on function reconstruction problems. We provide a parameter construction and empirical verification of these slopes for 1D function approximation. From the first principles we discover, we make one stride and propose the ``Gated Quadratic Unit'' which has an even steeper $L(P)$ slope than the GLU and MLP. This opens the possibility of architecture design from first principles numerical theory to unlock superior scaling in large models. Replication code is available at https://github.com/afqueiruga/divine_scaling.

Figures (9)

• Figure 1: Illustration of the function space of randomly initialized MLPs and GLUs with ReLU activations. Left and middle: randomly initialized MLP and GLU on $\mathbb{R}^2\rightarrow\mathbb{R}$. Black boundaries are the activation boundaries that break the domain into piecewise linear (MLP) or piecewise quadratic partitions. Right: in 1D, each neuron of the GLU forms a single piecewise quadratic function.
• Figure 2: Comparison of MLP and GLU fitting behavior under spline-based initialization (a,d) and full training (b, e). Vertical lines denote cell boundaries formed by $Gx+g=0$. Partial optimization in (a,d) already achieves the same asymptotic scaling rate as the analytical construction. The activations of individual neurons of the fully trained MLP and GLU are shown in (c,f).
• Figure 3: Experimental results for $L(P)$ scaling on 1d function approximation. The x-axis on the left uses neuron count which corresponds to the number of spline knots; changing the variable to parameter count on the right translates the log-log lines but does not change the log-log slope.
• Figure 4: MLPs and GLUs have similar approximation errors and order of convergence as their spline counterparts. Neuron count is analogous to the number of knots (control nodes) in a spline.
• Figure 5: Novel architecture design to scale faster than a GLU: the Gated Quadratic Unit (GQU) has faster scaling than a GLU at $L(P)\propto P^{-3.5}$.