Extracting and Understanding the Superficial Knowledge in Alignment

TL;DR

This work investigates whether alignment of large language models is largely superficial. It formalizes superficial knowledge as easily restylable patterns and isolates it by constraining changes to a shallow linear projection head, using distillation to preserve alignment signals. Empirically, superficial knowledge accounts for a large fraction of alignment, especially for safety and detoxification tasks, but deeper reasoning remains necessary for mathematics and truthful QA. The study further demonstrates practical benefits: superficial knowledge can be transferred across models for offsite alignment and recovered after disruptions to safety, offering a path to more scalable and robust alignment strategies while highlighting the need to address non-superficial aspects for full reliability.

Abstract

Alignment of large language models (LLMs) with human values and preferences, often achieved through fine-tuning based on human feedback, is essential for ensuring safe and responsible AI behaviors. However, the process typically requires substantial data and computation resources. Recent studies have revealed that alignment might be attainable at lower costs through simpler methods, such as in-context learning. This leads to the question: Is alignment predominantly superficial? In this paper, we delve into this question and provide a quantitative analysis. We formalize the concept of superficial knowledge, defining it as knowledge that can be acquired through easily token restyling, without affecting the model's ability to capture underlying causal relationships between tokens. We propose a method to extract and isolate superficial knowledge from aligned models, focusing on the shallow modifications to the final token selection process. By comparing models augmented only with superficial knowledge to fully aligned models, we quantify the superficial portion of alignment. Our findings reveal that while superficial knowledge constitutes a significant portion of alignment, particularly in safety and detoxification tasks, it is not the whole story. Tasks requiring reasoning and contextual understanding still rely on deeper knowledge. Additionally, we demonstrate two practical advantages of isolated superficial knowledge: (1) it can be transferred between models, enabling efficient offsite alignment of larger models using extracted superficial knowledge from smaller models, and (2) it is recoverable, allowing for the restoration of alignment in compromised models without sacrificing performance.

Figures (4)

• Figure 1: We extract superficial knowledge from an aligned model into a shallow linear projection head. The upper left shows the potential advantages brought by the extracted superficial knowledge, and the upper right shows the WordCloud of source shift tokens and target shift tokens, which primarily involves stylistic words.
• Figure 2: KL divergence between the base model and the aligned model, and between the base model with superficial knowledge and the aligned model
• Figure 3: Trade-off between informativeness(Shift token cover rate) and transferability(Transfer space similarity) with diffrent $k$
• Figure 4: K vs Token Accuracy