Theoretical Foundations of Scaling Law in Familial Models

TL;DR

Traditional neural scaling laws assume a single dense output and fail to account for Familial Models that generate $G$ deployable sub-models from one backbone. We propose a unified scaling law $L(N,D,G)$ and fit it under fixed compute budgets using an IsoFLOP design, yielding $L(N,D,G) = \left( E + \frac{A}{N^{\alpha}} + \frac{B}{D^{\beta}} \right) G^{\gamma}$ with $\gamma \approx 0.041$. The results show the granularity penalty is negligible, demonstrating that one training run can yield multiple models without compromising the compute-optimal frontier. This enables flexible deployment across heterogeneous device-edge-cloud hierarchies and motivates broader adoption of relay-style inference and collaborative inference workflows.

Abstract

Neural scaling laws have become foundational for optimizing large language model (LLM) training, yet they typically assume a single dense model output. This limitation effectively overlooks "Familial models, a transformative paradigm essential for realizing ubiquitous intelligence across heterogeneous device-edge-cloud hierarchies. Transcending static architectures, familial models integrate early exits with relay-style inference to spawn G deployable sub-models from a single shared backbone. In this work, we theoretically and empirically extend the scaling law to capture this "one-run, many-models" paradigm by introducing Granularity (G) as a fundamental scaling variable alongside model size (N) and training tokens (D). To rigorously quantify this relationship, we propose a unified functional form L(N, D, G) and parameterize it using large-scale empirical runs. Specifically, we employ a rigorous IsoFLOP experimental design to strictly isolate architectural impact from computational scale. Across fixed budgets, we systematically sweep model sizes (N) and granularities (G) while dynamically adjusting tokens (D). This approach effectively decouples the marginal cost of granularity from the benefits of scale, ensuring high-fidelity parameterization of our unified scaling law. Our results reveal that the granularity penalty follows a multiplicative power law with an extremely small exponent. Theoretically, this bridges fixed-compute training with dynamic architectures. Practically, it validates the "train once, deploy many" paradigm, demonstrating that deployment flexibility is achievable without compromising the compute-optimality of dense baselines.

Figures (2)

• Figure 1: Three-dimensional visualization of the fitted familial model scaling law. The horizontal axes represent the model size $N$ and training token count $D$ (both in log scale), while the vertical axis shows the fitted loss $L$. The surface reveals a smooth decay in loss as scale increases. The slight upward variation across granularity levels $G$ reflects the minimal multiplicative penalty encoded by the small exponent $\gamma \approx 0.041$, indicating that Familial Models maintain near-optimal scaling behavior.
• Figure 2: Compute-efficient frontier for granularity $G=3$. The plot displays the isoloss contours and the implied efficiency frontier (blue line) across varying FLOPs budgets. It indicates that moderate increases in training tokens $D$ can compensate for smaller model sizes $N$ to maintain constant loss.