Frac-Connections: Fractional Extension of Hyper-Connections

TL;DR

Frac-Connections introduce a memory-efficient fractional extension of Hyper-Connections by partitioning hidden states into $m=1/n$ fractions, enabling multiple connection strengths without widening activations. This dynamic/static framework maintains expressiveness while reducing memory access and computational overhead, with initialization and normalization strategies ensuring stable training. Empirical results on large-scale pretraining with dense and Mixture-of-Experts transformers show improved training stability and downstream performance across diverse NLP benchmarks, particularly when training to trillions of tokens. The approach offers a scalable, practical enhancement for next-generation transformers in both dense and sparse settings.

Abstract

Residual connections are central to modern deep learning architectures, enabling the training of very deep networks by mitigating gradient vanishing. Hyper-Connections recently generalized residual connections by introducing multiple connection strengths at different depths, thereby addressing the seesaw effect between gradient vanishing and representation collapse. However, Hyper-Connections increase memory access costs by expanding the width of hidden states. In this paper, we propose Frac-Connections, a novel approach that divides hidden states into multiple parts rather than expanding their width. Frac-Connections retain partial benefits of Hyper-Connections while reducing memory consumption. To validate their effectiveness, we conduct large-scale experiments on language tasks, with the largest being a 7B MoE model trained on up to 3T tokens, demonstrating that Frac-Connections significantly outperform residual connections.

Figures (8)

• Figure 1: Comparison of Frac-Connections and Hyper-Connections based on their expansion rates. Frac-Connections correspond to $n \leq 1$, while Hyper-Connections are defined by $n \geq 1$. The two connection types become identical when the expansion rate is $n = 1$.
• Figure 2: Cosine similarity between the input of the current and the previous layers for the OLMoE-7B models. The curve represents the median of similarity, while the shaded area indicates the range between the 5th and 95th percentiles.
• Figure 3: Figure 2. Frac-connections (FC) with an expansion rate of $n = 1/2$. (a) Residual connections. (b) Hyper-connections: $\beta_1$, $\beta_2$, $\alpha_{0,0}$, $\alpha_{0,1}$, $\alpha_{1,0}$, $\alpha_{1,1}$, $\alpha_{2,1}$, and $\alpha_{2,2}$ are learnable scalars or scalars predicted by the network, depending on the specific HC version. (c) Frac-connections: Frac-connections split the hidden representations into smaller fractions and process each fraction independently. The scalars $\gamma_{1,2}$, $\gamma_{2,1}$, and $\gamma_{2,2}$ are either learnable or predicted by the network, similar to hyper-connections. These fractions are concatenated (denoted as $\text{Cat}$) after processing, followed by integration into the main network pipeline.
• Figure 4: Training loss (0.999 EMA smoothed) loss for OLMoE-1.3B models.
• Figure 5: Training and evaluation performance of OLMoE-7B models. The plots show the training loss, C4-en loss, and accuracy on HellaSwag, SciQ, Commonsense QA, Social IQA, and WinoGrande over the course of training. The results are EMA-smoothed for clarity. The OLMoE-7B-DFCx4 variant demonstrates improved loss reduction and higher accuracy across multiple benchmarks compared to the baseline OLMoE-7B model, indicating enhanced optimization efficiency and generalization.