Revisiting Residual Connections: Orthogonal Updates for Stable and Efficient Deep Networks

TL;DR

This work revisits residual connections and argues that standard additive updates can excessively modulate the current residual stream. It proposes Orthogonal Residual Update, decomposing the module output into $f_{\parallel}$ and $f_{\perp}$ and updating with the orthogonal component $f_{\perp}$, i.e., $x_{n+1} = x_n + f_{\perp}(x_n)$. Across ViT and ResNetV2 on CIFAR, TinyImageNet, and ImageNet-1k, the method yields improved generalization and training stability, exemplified by a +3.78 percentage-point gain for ViT-B on ImageNet-1k, with modest computational overhead. The work analyzes internal dynamics, ablations, and robustness to hyperparameters, and discusses implications for residual-stream geometry and future research across architectures and modalities.

Abstract

Residual connections are pivotal for deep neural networks, enabling greater depth by mitigating vanishing gradients. However, in standard residual updates, the module's output is directly added to the input stream. This can lead to updates that predominantly reinforce or modulate the existing stream direction, potentially underutilizing the module's capacity for learning entirely novel features. In this work, we introduce Orthogonal Residual Update: we decompose the module's output relative to the input stream and add only the component orthogonal to this stream. This design aims to guide modules to contribute primarily new representational directions, fostering richer feature learning while promoting more efficient training. We demonstrate that our orthogonal update strategy improves generalization accuracy and training stability across diverse architectures (ResNetV2, Vision Transformers) and datasets (CIFARs, TinyImageNet, ImageNet-1k), achieving, for instance, a +3.78 pp top-1 accuracy gain for ViT-B on ImageNet-1k.

Figures (12)

• Figure 1: Intuition behind our orthogonal residual update.Left: The standard residual update adds the full output of module $f(x_n)$ to the input stream $x_n$. Right: Our proposed update first decomposes the module output $f(x_n)$ into a component parallel to $x_n$ ($f_{\parallel}$) and a component orthogonal to $x_n$ ($f_{\perp}$). We then discard $f_{\parallel}$ and add only the orthogonal component $f_{\perp}$ to the stream.
• Figure 2: Orthogonal update accelerates convergence and enhances generalization efficiency compared to the standard linear update baseline (ViT-B on ImageNet-1k results shown). (a) Faster Convergence: Orthogonal update (blue) achieves significantly lower training loss in fewer iterations. (b) Improved Time-to-Accuracy: Orthogonal update (blue) attains higher validation Acc@1 consistently outperforms linear update with minimal overhead.
• Figure 3: Internal dynamics (ViT-S, TinyImageNet, 5 seeds). Each subfigure shows blocks 0--5 (MLP top, Attention bottom). Ours denotes orthogonal updates; Linear denotes the standard residual. (a,c) After the Transition Point, orthogonal updates stabilize the stream norm $\|x_n\|^2$, whereas linear updates typically exhibit a post-transition decrease. (b,d) The parallel component energy $\|f_{\parallel}(x_n)\|^2$ follows distinct layer-wise profiles for linear vs. orthogonal updates. Signed parallel coefficients and orthogonal-component traces are analyzed in the Appendix \ref{['sec:detail_internal_norm']}.
• Figure 4: Learned stream–scaling variant: per-block trajectories of $\alpha_\ell$ (median across seeds; band = interquartile range). Values below zero correspond to stream attenuation. This diagnostic contrasts a layerwise, input-invariant scalar with our input-dependent projection.
• Figure 5: Effect of orthogonal update probability $\pi$ on ViT-S performance on TinyImageNet (N=3). Error bars represent $\pm$1 standard deviation. The Pearson correlation coefficient ($R$) and its p-value between $\pi$ and accuracy are displayed in each subplot, indicating a positive correlation.