Emergent inabilities? Inverse scaling over the course of pretraining

TL;DR

This study investigates whether inverse scaling extends from model size to the amount of pretraining data, exploring if performance on specific tasks degrades as training data accumulates. Using eight Pythia models (70M–12B) trained on The Pile (~300B tokens) and evaluated at eight checkpoints with the Language Model Evaluation Harness, the authors examine 12 tasks including TruthfulQA variants and Inverse Scaling Prize items. They find clear inverse scaling on five tasks (TruthfulQA-MC1, TruthfulQA-MC2, Hindsight Neglect, Memo Trap, Pattern Match Suppression) and possible evidence on additional tasks, with larger models exhibiting greater degradation as training proceeds; some tasks show no consistent trend. The results highlight the fragility of performance as training data scales and underscore the importance of continuous, task-specific evaluation during model updates to avoid overtrusting improvements on established benchmarks.

Abstract

Does inverse scaling only occur as a function of model size, or can it also occur over the course of training? We carry out an exploratory study investigating whether the performance of language models on specific tasks can decrease (while general performance remains high) during training on the language modeling task. We find 8 tasks on which Pythia 12B (Biderman et al., 2023) shows decreased performance over the course of training. Five of these tasks (TruthfulQA-MC1, TruthfulQA-MC2, Hindsight Neglect, Memo Trap, and Pattern Match Suppression) additionally show a consistent relationship whereby larger language models show a greater decrease in performance the more they are trained, despite showing standard (positive) scaling overall. This highlights the importance of testing performance at all relevant benchmarks any time models are trained on additional data, even if their overall performance improves

Figures (1)

• Figure 1: Performance of the 8 Pythia biderman_2023_PythiaSuiteAnalyzing models at 8 stages over the course of training at the two multiple-choice variants of TruthfulQAlin_2022_TruthfulQAMeasuringHow and the 10 multiple-choice winners of the Inverse Scaling Prize mckenzie_2023_InverseScalingWhen.