Virtual Width Networks

TL;DR

Virtual Width Networks (VWN) address the resource challenge of widening Transformer representations by decoupling embedding width from backbone width, enabling large increases in representational capacity with near-constant compute. The framework combines Over-Width Embedding, Generalized Hyper-Connections (GHC) and its dynamic variant (DGHC), plus Multi-Token Prediction (MTP) to exploit wider token spaces with efficient routing and supervision. Empirical results on Mixture-of-Experts models show substantial token efficiency gains, including a log-linear scaling relation between virtual width and loss, and consistent downstream improvements as width scales from 1.5× to 8×. While practical deployment carries hardware considerations, VWN provides a new dimension for scaling large models that complements depth, data, and standard width scaling, offering a principled path to improved quality-per-compute trade-offs.

Abstract

We introduce Virtual Width Networks (VWN), a framework that delivers the benefits of wider representations without incurring the quadratic cost of increasing the hidden size. VWN decouples representational width from backbone width, expanding the embedding space while keeping backbone compute nearly constant. In our large-scale experiment, an 8-times expansion accelerates optimization by over 2 times for next-token and 3 times for next-2-token prediction. The advantage amplifies over training as both the loss gap grows and the convergence-speedup ratio increases, showing that VWN is not only token-efficient but also increasingly effective with scale. Moreover, we identify an approximately log-linear scaling relation between virtual width and loss reduction, offering an initial empirical basis and motivation for exploring virtual-width scaling as a new dimension of large-model efficiency.

Figures (9)

• Figure 1: Results from large-scale experiments on a 3.3B-activation MoE using Virtual Width Networks (VWN). We compare the baseline MoE-A3.3B against MoE-A3.3B-VWNx8, configured with a virtual width factor of $r{=}8$. Left and middle: training loss for next-token and next-two-token prediction versus seen tokens. VWN reaches the same loss as the baseline using $2.5\times$ and $3.5\times$ fewer tokens, respectively. Right: average accuracy on a collection of open-source benchmarks (see Table \ref{['tab:benchmarks_b']}), where scores are aggregated using internally defined task weights. A difference of one point corresponds to a notable performance gap under this weighting scheme.
• Figure 2: Standard Transformer vs. Virtual Width Network (VWN). (a) A standard Transformer uses the same width for embeddings and backbone. (b) Naive width scaling expands both proportionally, causing quadratic growth in parameters and compute. (c) VWN decouples embedding width from backbone width. With Generalized Hyper‑Connections, over‑width embeddings (e.g., 1.5$\times$) are coupled to a standard‑width backbone, increasing representational capacity with minimal compute overhead.
• Figure 3: Overview of Virtual Width Networks (VWN). (a) The standard Transformer maintains a consistent width across input embeddings, intermediate hidden vectors at each layer, and final layer outputs. (b) VWN scales the embedding dimension through over-width embeddings while maintaining the layer dimension using lightweight Generalized Hyper-Connections (GHC). These dimensions interact flexibly through small matrices $\mathbf{A}^l$ and $\mathbf{B}^l$ ($l$ stands for the layer number). (c) We enable multiple token supervision (multi-token prediction), allowing for richer token representations.
• Figure 4: Performance of VWN and MTP on 0.4B/4B MoE models. Left: Training loss versus seen tokens (billions). VWN lowers the next-token prediction loss, whereas MTP slightly hurts the NTP loss; combining VWN and MTP (VWN+MTP) yields the lowest final loss among the augmented variants but still shows a small gap ( 0.016) relative to the baseline metric when MTP is included. Right: Average downstream accuracy (%) versus tokens. Both VWN and MTP improve downstream accuracy over the baseline, and their combination delivers the largest gains throughout training. Models: MoE-0.4B/4B (baseline), MoE-0.4B/4B-VWN, MoE-0.4B/4B-MTP, and MoE-0.4B/4B-VWN-MTP.
• Figure 5: Performance of VWN and MTP on 2.5B/25B MoE models. Left: Training loss versus seen tokens (billions). VWN reduces the next‑token prediction loss relative to the baseline, and adding MTP on top of VWN does not hurt the loss at this scale, with VWN+MTP reaching the lowest final loss, with a gap of 0.015 versus the baseline at the end of training. Right: Average downstream accuracy (%) versus tokens. Both VWN and VWN+MTP outperform the baseline, and VWN+MTP delivers the highest accuracy throughout training. Models: MoE-2.5B/25B (baseline), MoE-2.5B/25B-VWN, and MoE-2.5B/25B-VWN-MTP.