Effective Frontiers: A Unification of Neural Scaling Laws

TL;DR

This paper introduces Effective Frontier theory to unify neural scaling laws across model capacity, data, and compute. By modeling learning as progressively advancing a frontier into a Zipfian tail, it derives explicit power-law scalings: $\Delta L(N) \asymp N^{-\gamma(\alpha-1)}$, $\Delta L(D) \asymp D^{-(\alpha-1)/\alpha}$, and $\Delta L(\tau) \asymp \tau^{-(\alpha-1)/(\alpha\beta)}$, with frontier positions $k_*(N) \propto N^{\gamma}$, $k_*(D) \propto D^{1/\alpha}$, and $k_*(\tau) \propto \tau^{1/(\alpha\beta)}$. A Max-Bottleneck composition principle reconciles Kaplan and Chinchilla as equilibrium solutions under different active bottlenecks, while a self-similar compute kernel extends the results to general optimizers. Empirical synthetic experiments validate sharp frontiers, exponent universality across tail indices, and a robust estimated bias $\beta \approx 2$. The framework provides a principled, geometry-driven bridge between data structure and optimization dynamics, with practical implications for data pruning and curriculum learning to accelerate scaling.

Abstract

Neural scaling laws govern the prediction power-law improvement of test loss with respect to model capacity ($N$), datasize ($D$), and compute ($C$). However, existing theoretical explanations often rely on specific architectures or complex kernel methods, lacking intuitive universality. In this paper, we propose a unified framework that abstracts general learning tasks as the progressive coverage of patterns from a long-tail (Zipfian) distribution. We introduce the Effective Frontier ($k_\star$), a threshold in the pattern rank space that separates learned knowledge from the unlearned tail. We prove that reducible loss is asymptotically determined by the probability mass of the tail a resource-dependent frontier truncation. Based on our framework, we derive the precise scaling laws for $N$, $D$, and $C$, attributing them to capacity, coverage, and optimization bottlenecks, respectively. Furthermore, we unify these mechanisms via a Max-Bottleneck principle, demonstrating that the Kaplan and Chinchilla scaling laws are not contradictory, but equilibrium solutions to the same constrained optimization problem under different active bottlenecks.

Figures (7)

• Figure 1: The Effective Frontier in Rank Space. The residual profile $q_k$ vs. pattern rank $k$ (log scale) under varying constraints: (Left) Model Capacity $N$, (Middle) Dataset Size $D$, and (Right) Compute $\tau$ (noting that $C \propto \tau$ for a fixed model configuration). In all cases, increasing resources pushes a sharp coverage frontier $k_\star$ deeper into the Zipfian tail ($\alpha=1.5$). This validates the geometric abstraction (Definition \ref{['def:effective_frontier']}) where learning is viewed as a progressive coverage process.
• Figure 2: Effective frontier $k_\star(R)$ in rank space. Under Zipf frequencies $p_k\propto k^{-\alpha}$, resources $R$ (e.g., $N,D,\tau$) induce a cutoff: learned patterns ($k\le k_\star$) vs. unlearned tail ($k>k_\star$). The reducible loss is dominated by the tail sum $\sum_{k>k_\star(R)} p_k$, which decreases as $k_\star(R)$ shifts right with increasing resources.
• Figure 3: Tail Heaviness Governs Data Scaling Efficiency.(a) The data scaling exponent $\alpha_D$ is analytically determined by the tail index $\alpha$ (Theorem \ref{['thm:data_scaling']}). Lighter tails (larger $\alpha$, e.g., via data pruning) yield strictly higher sample efficiency. (b) Geometrically, for a fixed effective frontier $k_*$, heavy-tailed distributions (Red) retain significantly more unlearned probability mass (shaded tail) than curated distributions (Teal), resulting in slower loss reduction.
• Figure 4: Geometric Scaling of the Frontier. We extract $k_\star$ (at threshold $\delta=0.5$) as a function of resources. (Left) Capacity frontier $k_\star \propto N^{0.53}$. (Middle) Coverage frontier $k_\star \propto D$. Crucially, the theoretical prediction $D^{1/\alpha}$ (thin dash) perfectly matches the empirical fit (thick dash) for $\alpha=1.5$, confirming Theorem \ref{['thm:data_scaling']}. (Right) Optimization frontier $k_\star \propto \tau$. Since total compute $C \propto \tau$ for a fixed model, this reflects the compute scaling law. The slope allows us to estimate the optimization bias $\beta \approx 2.03$.
• Figure 5: Universal Scaling Laws. Reducible loss $\Delta L$ vs. resources for distributions with different tail heaviness $\alpha$. (Center) For Data Scaling, the empirical curves (solid lines) align perfectly with the theoretical predictions $\Delta L \propto D^{-(\alpha-1)/\alpha}$ (dashed lines), confirming that $\alpha$ dictates sample efficiency. (Right) Compute scaling laws (measured via steps $\tau$, where $C \propto \tau$ for fixed models) show that lighter tails (larger $\alpha$) lead to faster convergence, consistent with the derived exponent $-\frac{\alpha-1}{\alpha\beta}$.

Theorems & Definitions (22)

• Definition 3.2: Effective Frontier
• Theorem 3.3: Universal Scaling Principle
• Remark 4.2
• Proposition 4.3: Model Scaling Law
• Definition 5.1: Residual Proxy
• Theorem 5.2: Data Scaling Law
• Definition 5.3: Generalized Residual Proxy
• Corollary 5.4
• Lemma 6.2: Iterative Residual Dynamics
• Remark 6.3