Zero-Shot Performance Prediction for Probabilistic Scaling Laws

TL;DR

This paper tackles the challenge of predicting NLP learning curves and probabilistic scaling laws under data and compute constraints by framing LC prediction as a multitask problem with a bi‑level hierarchy modeled by latent variable multi‑output Gaussian Processes (MaGP). It derives a predictive framework that yields zero‑shot LC predictions and Monte Carlo‑based probabilistic scaling laws, augmented by an active learning strategy to reduce predictive uncertainty. Across nanoGPT, bilingual, and multilingual translation datasets, MaGP consistently outperforms baselines such as DHGP and Bayesian nets, while demonstrating that the two‑level hierarchies are exchangeable and useful for cross‑task transfer. The approach offers a cost‑efficient path to estimating scaling behavior and planning data collection and model selection, with practical implications for architecture design and multilingual NLP systems.

Abstract

The prediction of learning curves for Natural Language Processing (NLP) models enables informed decision-making to meet specific performance objectives, while reducing computational overhead and lowering the costs associated with dataset acquisition and curation. In this work, we formulate the prediction task as a multitask learning problem, where each task's data is modelled as being organized within a two-layer hierarchy. To model the shared information and dependencies across tasks and hierarchical levels, we employ latent variable multi-output Gaussian Processes, enabling to account for task correlations and supporting zero-shot prediction of learning curves (LCs). We demonstrate that this approach facilitates the development of probabilistic scaling laws at lower costs. Applying an active learning strategy, LCs can be queried to reduce predictive uncertainty and provide predictions close to ground truth scaling laws. We validate our framework on three small-scale NLP datasets with up to $30$ LCs. These are obtained from nanoGPT models, from bilingual translation using mBART and Transformer models, and from multilingual translation using M2M100 models of varying sizes.

Figures (26)

• Figure 1: A task from each dataset illustrating the modelled two-layer hierarchy. (left) Learning curves assuming common prior information when the same number of embedding parameters is used. (middle) Learning curves for bilingual translation, assuming common prior information when translating from the same source language. (right) Learning curves for multilingual translation, assuming common prior information when the source language and model size are the same.
• Figure 2: A set of learning curves from nanoGPT models, each showing loss versus compute for different model sizes. Larger models achieve lower test loss.
• Figure 3: The nanoGPTlarge dataset. LCs for different combinations of embedding parameters and layers are shown, with the minimum achieved loss indicated above each curve.
• Figure 4: (left) Bilingual learning curve dataset. No data is available on the main diagonal. Light blue squares indicate test-set performance curves. Dashed lines: learning curves from the fine-tuned mBART50 model. Solid lines: learning curves from the Transformer model trained from scratch. (right) Multilingual learning curve dataset. We perform few-shot extrapolation for the M2M$100$-$1.2$B model, translating from Indonesian (id) into multiple target languages. The $615$M and $175$M models correspond to two separate tasks under this assumption. Light blue sections of the curves mark the extrapolated data points.
• Figure 5: Schematic showing the train-test splits of the nanoGPTlarge dataset: Quad (top) and Tri (bottom). Blue occluded learning curves are the test data.