Neuroplasticity and Corruption in Model Mechanisms: A Case Study Of Indirect Object Identification

TL;DR

The paper investigates how task-specific fine-tuning and data poisoning reshape the mechanistic circuitry underlying Indirect Object Identification in a GPT-2-small model. It applies mechanistic interpretability tools, including Path Patching, Knockout, and Cross-Model Activation Patching (CMAP), to analyze circuit changes across clean and corrupted fine-tuning, and then examines neuroplasticity by retraining on clean data. Key findings show that fine-tuning amplifies existing mechanisms, corruption from toxic fine-tuning localizes to specific attention heads, and the model relearns original mechanisms after retraining, demonstrating robust neuroplasticity. These results have implications for safety and defense, highlighting how small, targeted retraining can restore intended behavior and how mechanistic insights can inform attack resilience.

Abstract

Previous research has shown that fine-tuning language models on general tasks enhance their underlying mechanisms. However, the impact of fine-tuning on poisoned data and the resulting changes in these mechanisms are poorly understood. This study investigates the changes in a model's mechanisms during toxic fine-tuning and identifies the primary corruption mechanisms. We also analyze the changes after retraining a corrupted model on the original dataset and observe neuroplasticity behaviors, where the model relearns original mechanisms after fine-tuning the corrupted model. Our findings indicate that: (i) Underlying mechanisms are amplified across task-specific fine-tuning which can be generalized to longer epochs, (ii) Model corruption via toxic fine-tuning is localized to specific circuit components, (iii) Models exhibit neuroplasticity when retraining corrupted models on clean dataset, reforming the original model mechanisms.

Figures (47)

• Figure 1: The new circuit we discovered for task-specific fine-tuning at Epoch 3. The emerging, marked in blue, circuit components formed performed similar mechanisms as the prior circuit components.
• Figure 2: Corrupted data augmentations we utilize to poison model behavior on task
• Figure 3: Attention Probability vs Projection of head output along $W_U[IO]$ and $W_U[S]$ for head L9H9
• Figure 4: \ref{['fig:ca_logit_attribution']}) Logit Attribution of heads L9H9, L11H10, L10H10 in original/amplified model. \ref{['fig:ca_logit_attribution_absolute']}) Absolute Logit Difference in the original model vs amplified model after ablation
• Figure 5: :Attention Probability vs Projection of head output along $W_U[IO]$ and $W_U[S]$ for head L10H10