Collapse of Self-trained Language Models

TL;DR

The paper evaluates the feasibility of self-training language models on their own outputs, using GPT-2 to test whether such loops can improve or degrade performance. It formalizes self-training as adapting to the local sequence distribution $P_l(x)$ via cross-entropy updates and evaluates the process on standard benchmarks. The key finding is that extended self-training induces degradation and rapid output collapse, accelerated by higher learning rates and larger model sizes, with similar patterns observed across datasets like Wikitext-2 and Penn Treebank. This work highlights critical risks of self-reinforcement in text-generation systems and calls for new architectures or safeguards to prevent uncontrollable degradation and data contamination in future iterative training regimes.

Abstract

In various fields of knowledge creation, including science, new ideas often build on pre-existing information. In this work, we explore this concept within the context of language models. Specifically, we explore the potential of self-training models on their own outputs, akin to how humans learn and build on their previous thoughts and actions. While this approach is intuitively appealing, our research reveals its practical limitations. We find that extended self-training of the GPT-2 model leads to a significant degradation in performance, resulting in repetitive and collapsed token output.

Figures (4)

• Figure 1: Schema of the self-training. The model generates a sequence $s_1$, computes its probability $P(s_1|\theta0_l)$, which is then used to determine the cross-entropy loss with gradient $\nabla\mathcal{L}(s_1)$ to update the next state of the model with the adapted parameters.
• Figure 2: Impact of learning rate on self-training GPT-2 gpt-2 language model on valid and train sets. As the learning rate increases, the model's performance deteriorates, leading to a higher loss on the valid set. On the train set, the model collapses and converges into a generation of repetitive tokens, resulting in almost zero loss on generated data. The y-axis represents the loss, and the x-axis displays the number of model steps.
• Figure 3: Correlation between model size and the onset of collapse in GPT-2 architectures.
• Figure 4: Comparative analysis of learning rate impact on self-trained GPT-2 model performance, evaluated on it's output (train loss) and validation subset of the Penn Treebank dataset.