ParaFormer: Shallow Parallel Transformers with Progressive Approximation

TL;DR

This work addresses the inefficiency of deep Transformer models by reframing depth as a product of inter-layer collaboration for progressive approximation rather than as a necessity for performance. It introduces ParaFormer, a shallow, parallel-branch Transformer that enforces progressive approximation through staged, inter-branch training, backed by a closed-form formulation and the Universal Approximation Theorem. The method yields competitive or superior accuracy to ViT on CIFAR-10, CIFAR-100, and Fashion-MNIST, while enabling up to $15.07×$ total compression through branch pruning and quantization, and achieving substantial inference speedups on multi-GPU deployments (about $3.30×$ vs FairScale/GPipe). These results demonstrate that depth-heavy architectures are not the only path to strong performance and open avenues for efficient Transformer designs with adaptive expansion capabilities.

Abstract

The widespread 'deeper is better' philosophy has driven the creation of architectures like ResNet and Transformer, which achieve high performance by stacking numerous layers. However, increasing model depth comes with challenges such as longer training times, higher inference latency, and impracticality on resource-constrained devices. To address these issues, we propose ParaFormer, a shallow Transformer architecture designed for true parallelism in both structure and computation. By formulating standard Transformers as function approximators in closed-form, our theoretical analysis shows that their performance relies on inter-layer collaboration for progressive approximation, rather than depth itself. While deep Transformers enforce this collaboration through sequential designs, we demonstrate that such collaboration is not inherently tied to sequential structures. ParaFormer removes the sequential constraint by organizing layers into parallel branches, enforcing inter-layer collaboration algorithmically. Specifically, we implement progressive approximation, ensuring that each new branch further reduces the loss from preceding branches, enabling faster convergence. Extensive experiments validate ParaFormer's effectiveness, outperforming standard Transformers like ViT. Moreover, ParaFormer supports up to 15.07x model compression and facilitates model expansion for adaptive continuous learning. Experimental results on multi-GPU deployment demonstrate that ParaFormer is 3.30x faster than widely used parallelism solutions such as FairScale. These advancements stem from our closed-form formulation of Transformers based on the Universal Approximation Theorem, which not only explains the ``depth belief'' but also opens new avenues for designing efficient Transformer architectures. Source code: https://(open-upon-acceptance)

Figures (7)

• Figure 1: Architectures of Transformer and its parallelism variants: (a) Original Transformer; (b) Model Parallelism; and (c) The Proposed ParaFormer.
• Figure 2: The $i^{th}$ layer of a Transformer illustrated using the notations defined in this paper.
• Figure 3: The progressive training process in ParaFormer: new branches are incrementally activated to promote broader inter-branch collaboration once previous branches have been sufficiently trained.
• Figure 4: The layer-wise accuracy and loss curves of ViT and ParaFormer on CIFAR-10, CIFAR-100 and Fashion-MNIST.
• Figure 5: Layer(Branch)-wise feature distributions of ViT$^{12}$ and ParaFormer$^1_{12}$ on CIFAR-10. ParaFormer$^1_{12}$ achieves satisfactory representations earlier than ViT$^{12}$.