Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

PTPP-Aware Adaptation Scaling Laws: Predicting Domain-Adaptation Performance at Unseen Pre-Training Budgets

Etienne Goffinet, Shane Bergsma, Avraham Sheinin, Natalia Vassilieva, Shaheer Muhammad, Preslav Nakov, Gurpreet Gosal

TL;DR

The paper introduces Ptpp-aware adaptation scaling laws that explicitly model the pre-training budget $P_{\mathrm{tpp}}$ as a driver of domain-adaptation performance. By proposing three formulations that combine a loss floor and data-exponent gating, the authors demonstrate accurate prediction of target-domain loss at unseen budgets, notably $P_{\mathrm{tpp}}=279$, in a multilingual English/Arabic to French setting and with various model sizes. A small set of 241M-scale anchors further improves calibration, and a practical use-case shows how the fitted laws enable optimized replay ratios and adaptation budgets under compute constraints. This work clarifies how pre-training compute modulates both the loss floor and adaptation efficiency, offering a scalable tool for budgeting, scheduling, and planning in continual pre-training and domain adaptation tasks.

Abstract

Continual pre-training (CPT) for domain adaptation must balance target-domain gains with stability on the base domain. Existing CPT scaling laws typically assume a fixed pre-training budget, which limits their ability to forecast adaptation outcomes for models trained at different tokens-per-parameter (PTPP). We present \emph{PTPP-aware} adaptation scaling laws that make the pre-training budget an explicit variable, enabling accurate \emph{prediction} of adaptation loss at unseen \ptpp. On a multilingual setup (English/Arabic $\rightarrow$ French), PTPP-aware formulations trained on early stages (\ptpp{}=\{15,31\}) predict target loss at \ptpp{}=279 and outperform a PTPP-agnostic \dcpt{} transfer baseline on metrics (Huber-on-log, MAE$_\mathrm{rel}$, calibration slope); full diagnostics (RMSE, MAPE) are in the appendix. Beyond forecasting, we show a practical use case: planning replay ratios and adaptation token budgets that satisfy target and forgetting constraints under compute limits.

PTPP-Aware Adaptation Scaling Laws: Predicting Domain-Adaptation Performance at Unseen Pre-Training Budgets

TL;DR

The paper introduces Ptpp-aware adaptation scaling laws that explicitly model the pre-training budget as a driver of domain-adaptation performance. By proposing three formulations that combine a loss floor and data-exponent gating, the authors demonstrate accurate prediction of target-domain loss at unseen budgets, notably , in a multilingual English/Arabic to French setting and with various model sizes. A small set of 241M-scale anchors further improves calibration, and a practical use-case shows how the fitted laws enable optimized replay ratios and adaptation budgets under compute constraints. This work clarifies how pre-training compute modulates both the loss floor and adaptation efficiency, offering a scalable tool for budgeting, scheduling, and planning in continual pre-training and domain adaptation tasks.

Abstract

Continual pre-training (CPT) for domain adaptation must balance target-domain gains with stability on the base domain. Existing CPT scaling laws typically assume a fixed pre-training budget, which limits their ability to forecast adaptation outcomes for models trained at different tokens-per-parameter (PTPP). We present \emph{PTPP-aware} adaptation scaling laws that make the pre-training budget an explicit variable, enabling accurate \emph{prediction} of adaptation loss at unseen \ptpp. On a multilingual setup (English/Arabic French), PTPP-aware formulations trained on early stages (\ptpp{}=\{15,31\}) predict target loss at \ptpp{}=279 and outperform a PTPP-agnostic \dcpt{} transfer baseline on metrics (Huber-on-log, MAE, calibration slope); full diagnostics (RMSE, MAPE) are in the appendix. Beyond forecasting, we show a practical use case: planning replay ratios and adaptation token budgets that satisfy target and forgetting constraints under compute limits.
Paper Structure (19 sections, 9 equations, 3 figures, 2 tables)

This paper contains 19 sections, 9 equations, 3 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Methodology and Experiments
  3. Setup.
  4. $\mathit{P}\mathrm{\textsc{tpp}}$-Aware candidate formulations (1–3).
  5. (1) Additive $\mathit{P}\mathrm{\textsc{tpp}}$ prior (floor).
  6. (2) $\mathit{P}\mathrm{\textsc{tpp}}$-gated data exponent (no floor).
  7. (3) $\mathit{P}\mathrm{\textsc{tpp}}$-gated data exponent + floor.
  8. Anchors.
  9. Data & models.
  10. Fitting constraints.
  11. Metrics.
  12. Results: Forecasting at Unseen $\mathit{P}\mathrm{\textsc{tpp}}$
  13. Takeaways.
  14. Use Case: Joint Compute and Replay Optimization
  15. Conclusion
  16. ...and 4 more sections

Figures (3)

  • Figure 1: $\mathit{P}\mathrm{\textsc{tpp}}$=279 predictions of the gated+floor model (dashed) vs. observations (markers) of validation loss for $r\in\{0.10,0.25,0.50\}$ and {241M, 517M, 1.4B, 8.1B}.
  • Figure 2: Replay (0–100%) determines the trade-off between forgetting and domain performance. Left: Forgetting / Dataset size landscape. Right: resulting French loss. The star highlights the solution (8.9 $\mathit{A}\mathrm{\textsc{tpp}}$, 34% replay), minimizing FLOPs s.t. forgetting is $\leq$+2% and French loss $\leq$ 1.8.
  • Figure 3: In-sample fits for Form 3 (gated+floor). Rows: $r\in\{0.10,0.25,0.50\}$; columns: {241M, 517M, 1.4B, 8.1B}. Dashed: fitted curves; markers: observations. Used only as an auxiliary fit-quality check.