Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Ablate and Rescue: A Causal Analysis of Residual Stream Hyper-Connections

William Peng, Josheev Rai, Kevin Tseng, Siwei Wang, Sean Wu

Abstract

Multi-stream transformer architectures have recently been proposed as a promising direction for managing representation collapse and the vanishing gradient problem for residual connections, yet their internal mechanisms remain unexplored. In particular, the recently introduced Manifold-Constrained Hyper-Connections (mHC) architecture posits multiple residual streams with constrained interaction, but lacks in-depth mechanistic analysis. We present the first open-source mHC language model (https://huggingface.co/wgpeng/mhc-780m) and analyze the multiple-stream architecture with a suite of representation-level metrics and causal interventions to probe how parallel streams encode and utilize information. Specifically, we introduce a systematic stream ablation-and-rescue framework that enables direct causal comparison of residual streams during inference. Through targeted pairwise interventions and controlled recovery experiments, we distinguish functional redundancy from asymmetric utilization and reveal how information is distributed across streams beyond what is observable from representational similarity alone.

Ablate and Rescue: A Causal Analysis of Residual Stream Hyper-Connections

Abstract

Multi-stream transformer architectures have recently been proposed as a promising direction for managing representation collapse and the vanishing gradient problem for residual connections, yet their internal mechanisms remain unexplored. In particular, the recently introduced Manifold-Constrained Hyper-Connections (mHC) architecture posits multiple residual streams with constrained interaction, but lacks in-depth mechanistic analysis. We present the first open-source mHC language model (https://huggingface.co/wgpeng/mhc-780m) and analyze the multiple-stream architecture with a suite of representation-level metrics and causal interventions to probe how parallel streams encode and utilize information. Specifically, we introduce a systematic stream ablation-and-rescue framework that enables direct causal comparison of residual streams during inference. Through targeted pairwise interventions and controlled recovery experiments, we distinguish functional redundancy from asymmetric utilization and reveal how information is distributed across streams beyond what is observable from representational similarity alone.
Paper Structure (24 sections, 4 equations, 9 figures, 2 tables)

This paper contains 24 sections, 4 equations, 9 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Background
  3. Hyper-connections and manifold constraint.
  4. Interpretability.
  5. Methods
  6. Model Training
  7. Centered Kernel Alignment
  8. Activation Patching
  9. Stream Ablation and Rescue
  10. Stream Ablation.
  11. Targeted Rescue.
  12. Results
  13. Representational Similarity Across Streams
  14. Stream-level Causal Contributions via Activation Patching
  15. Functional Redundancy and Asymmetry via Rescue
  16. ...and 9 more sections

Figures (9)

  • Figure 1: Ablation-and-rescue for causal stream analysis. (a) Counterfactual activation patching setup. (b) Ablation-and-rescue for multi-stream architectures.
  • Figure 2: Within-layer similarity.
  • Figure 3: Layer–stream causal sensitivity via activation patching. Mean KL divergence between baseline and patched logits when one (layer, stream) activation is injected from source to target run. Lighter values indicate stronger causal effect.
  • Figure 4: Within-layer CKA similarity matrices across depth. Middle layers show clear block structure, reflecting soft partitioning into redundant stream subgroups.
  • Figure 5: Inter-layer CKA with streamwise concatenation. Layers evolve gradually in their representational geometry.
  • ...and 4 more figures