Revisiting the Superficial Alignment Hypothesis

TL;DR

New light is shed on the Superficial Alignment Hypothesis by empirically studying the scaling behavior of post-training with increasing finetuning examples and evaluating them using objective task-specific standardized benchmarks, suggesting that it is, at best, an over-simplification.

Abstract

The Superficial Alignment Hypothesis posits that almost all of a language model's abilities and knowledge are learned during pre-training, while post-training is about giving a model the right style and format. We re-examine these claims by empirically studying the scaling behavior of post-training with increasing finetuning examples and evaluating them using objective task-specific standardized benchmarks. Through experiments with the Llama-3, Mistral, and Llama-2 model families of multiple sizes, we observe that, similar to the pre-training scaling laws, post-training task performance scales as a power law against the number of finetuning examples. This power law relationship holds across a broad array of capabilities, including mathematical reasoning, coding, instruction following, and multihop-reasoning. In addition, for tasks like math and multihop reasoning, we observe that a handful of examples merely align the model stylistically but do not saturate performance on the benchmarks. Model performance is instead correlated with its reasoning ability and it improves significantly with more examples, illustrating the need for holistic evaluation programs leveraging objective benchmarks in addition to measurement of alignment to human preferences. We also observe that language models are not necessarily limited to using knowledge learned during pre-training. With appropriate post-training, a model's ability to integrate new knowledge greatly improves on downstream tasks like multihop question-answering. Taken together, these results shed new light on the Superficial Alignment Hypothesis, suggesting that it is, at best, an over-simplification.

Figures (7)

• Figure 1: Performance improvements as finetuning data is scaled up, for models in the sub-10 Billion parameter range. The points are fitted with a power law curve of the form $P \propto D^{1 / b}$. Model performance consistently scales in a power law fashion, across model families.
• Figure 2: Performance scaling curves with increasing model size for models in the same family.
• Figure 3: Breakdown of error responses by models finetuned with datasets of increasing data scales. The first group in each chart shows the Total Mistakes made on the test set by the models. Each error response is then independently evaluated for the different mistake types and thus can belong to multiple error types. There is a clear trend of models saturating on style and formatting improvements with just a few examples. However, reasoning and arithmetic errors continue to get better.
• Figure 4: Example of an Event and Question Pairs from the curated Facts100 dataset.
• Figure 5: Error Analysis on the New Fact Multihop Questions after fine-tuning. BM stands for the pre-trained base model and PM for the multihop reasoning post-trained model.