Scaling Laws Meet Model Architecture: Toward Inference-Efficient LLMs

TL;DR

This work introduces architecture-aware scaling laws for large language models by explicitly incorporating architectural factors—hidden size, mlp-to-attention ratio, and grouped-query attention (GQA)—into a conditional extension of the Chinchilla framework. Through training and evaluating over 200 models from 80M to 3B parameters, the authors fit a conditional scaling law and develop a search framework to identify model architectures that maximize inference throughput without sacrificing accuracy. The method yields architectures that under the same budget achieve up to 2.1% higher downstream accuracy and up to 42% greater inference throughput than baselines like LLaMA-3.2, with validation at 1B and 3B scales and a demonstrated ability to predict optimal configurations. These results offer a practical pathway to inference-efficient LLM deployment and broaden the applicability of scaling laws beyond sheer parameter/data scaling to include architectural design spaces.

Abstract

Scaling the number of parameters and the size of training data has proven to be an effective strategy for improving large language model (LLM) performance. Yet, as these models grow increasingly powerful and widely deployed, the cost of inference has become a pressing concern. Despite its importance, the trade-off between model accuracy and inference efficiency remains underexplored. In this work, we examine how key architectural factors, hidden size, the allocation of parameters between MLP and attention (mlp-to-attention ratio), and grouped-query attention (GQA), influence both inference cost and accuracy. We introduce a conditional scaling law that augments the Chinchilla framework with architectural information, along with a search framework for identifying architectures that are simultaneously inference-efficient and accurate. To validate our approach, we train more than 200 models spanning 80M to 3B parameters and 8B to 100B training tokens, and fit the proposed conditional scaling law. Our results show that the conditional scaling law reliably predicts optimal architectural choices and that the resulting models outperform existing open-source baselines. Under the same training budget, optimized architectures achieve up to 2.1% higher accuracy and 42% greater inference throughput compared to LLaMA-3.2.

Figures (17)

• Figure 1: Although larger models generally achieve lower inference throughput than smaller ones, Qwen2.5-1.5B outperforms Qwen3-0.6B. Despite having the same number of layers, Qwen2.5-1.5B benefits from a higher hidden size, GQA, and mlp-to-attention ratio.
• Figure 2: Inference throughput vs (left) hidden size $d = d_\text{model}$ and (right) mlp-to-attention ratio $r = r_{\text{mlp}/\text{attn}}$ on the 8B model. Under a fixed parameter budget $N_{\text{non-embed}}$, larger hidden sizes and higher mlp-to-attention ratios improve inference throughput for varying batch sizes.
• Figure 3: Loss vs. hidden size: (Left) 80M model variants; (Center) 145M model variants; (Right) 297M model variants. Across model sizes, the relationship between training loss and $d_\text{model} / \sqrt{N}$ exhibits a consistent U-shaped curve when architectural factors such as GQA and the MLP-to-attention ratio are held fixed. The legend denotes the MLP-to-attention ratio $r = r_{\text{mlp}/\text{attn}}$ for each model.
• Figure 4: Loss vs. MLP-to-attention ratio: (Left) 80M model variants; (Center) 145M model variants; (Right) 297M model variants. Across model sizes, the relationship between training loss and $r_{\text{mlp}/\text{attn}}$ exhibits a consistent U-shaped curve when architectural factors such as GQA and hidden size are held fixed. The legend denotes the hidden size $d = d_\text{model}$ for each model.
• Figure 5: Predictive performances of the fitted conditional scaling law on: (left) Task 1: Fit on $80$M, evaluate on $145$M; (center) Task 2: Fit on $80, 145$M, evaluate on $297$M; (right) Task 3: Fit on $80, 145, 297$M, evaluate on $1$B. Orange dots denote fitting data points, and purple crosses indicate the test data points. We compare scaling-law predicted loss with actual pretraining loss of architectures and observed a consistently low MSE and high Spearman correlation across model scales.