Parallel Scaling Law for Language Models

TL;DR

The paper introduces ParScale, a parallel-scaling paradigm that reuses existing parameters to run multiple input variants in parallel and dynamically aggregates their outputs. It provides both theoretical and empirical support, showing that P parallel streams approximate parameter scaling by about O(log P) while reducing memory and latency costs, and demonstrates this through large-scale pretraining on Stack-V2 and Pile with up to 8 streams. A practical, logarithmic scaling law is fitted and validated, and a two-stage training strategy is proposed to mitigate training costs. The approach is extended to off-the-shelf models and instruction tuning, with notable gains on reasoning-intensive tasks and promising applicability to edge devices and other domains.

Abstract

It is commonly believed that scaling language models should commit a significant space or time cost, by increasing the parameters (parameter scaling) or output tokens (inference-time scaling). We introduce the third and more inference-efficient scaling paradigm: increasing the model's parallel computation during both training and inference time. We apply $P$ diverse and learnable transformations to the input, execute forward passes of the model in parallel, and dynamically aggregate the $P$ outputs. This method, namely parallel scaling (ParScale), scales parallel computation by reusing existing parameters and can be applied to any model structure, optimization procedure, data, or task. We theoretically propose a new scaling law and validate it through large-scale pre-training, which shows that a model with $P$ parallel streams is similar to scaling the parameters by $O(\log P)$ while showing superior inference efficiency. For example, ParScale can use up to 22$\times$ less memory increase and 6$\times$ less latency increase compared to parameter scaling that achieves the same performance improvement. It can also recycle an off-the-shelf pre-trained model into a parallelly scaled one by post-training on a small amount of tokens, further reducing the training budget. The new scaling law we discovered potentially facilitates the deployment of more powerful models in low-resource scenarios, and provides an alternative perspective for the role of computation in machine learning.

Figures (7)

• Figure 2: Loss of LLMs scaled on parameters and number of parallel streams $P$ trained on 42B tokens. Each point depicts the loss from a training run. The fitted scaling law curve from \ref{['eq:parscale_law_practical']} is displayed, with annotated fitted parameters $(E,A,k,\alpha)$ and the goodness of fit $R^2$.
• Figure 3: Predicted loss contours for ParScale. Each contour line indicates a combination of (parameter, $P$) with similar performance.
• Figure 4: Model capacity (indicated by loss) scales on the inference space-time cost, with three parameters (1.6B, 2.8B, and 4.4B) and batch sizes $\in\{1, 2, 4, 8\}$. Results are averaged from input / output tokens $\in\{64,128,256,512\}$. Blue arrows indicate parameter scaling; gray arrows represent parallel scaling.
• Figure 5: Loss for two-stage training, smoothing using an exponential moving average with a weight of 0.95.
• Figure 6: (a)(b) Loss for continual pre-training the Qwen-2.5-3B model on the two datasets. (c) Code generation performance after fine-tuning on Stack-V2 (Python), averaged from HumanEval(+) and MBPP(+). We only fine-tune the introduced parameters (prefix tokens and aggregation weights), with different $P$ sharing exactly the same Qwen2.5-3B pre-trained weights.

Theorems & Definitions (3)

• Lemma 3.1: Chinchilla Scaling Law chinchilla
• Proposition 1: Theoretical Formula for Parallel Scaling Law
• proof