Universal Neural Architecture Space: Covering ConvNets, Transformers and Everything in Between

TL;DR

The paper tackles the limitation that NAS often fails to outperform hand-crafted architectures and is constrained by narrow benchmarks. It introduces UniNAS, a universal graph-based search space that unifies CNNs, transformers, and hybrids, along with a training-free proxy and a standardized toolkit for reproducible NAS. An architecture search algorithm navigates the UniNAS space to discover novel topologies, including UniNAS-A, which surpasses hand-crafted baselines under identical training conditions. The work demonstrates strong performance on ImageNet and downstream tasks, and provides a publicly available toolkit to foster fair comparisons and broader adoption in NAS research.

Abstract

We introduce Universal Neural Architecture Space (UniNAS), a generic search space for neural architecture search (NAS) which unifies convolutional networks, transformers, and their hybrid architectures under a single, flexible framework. Our approach enables discovery of novel architectures as well as analyzing existing architectures in a common framework. We also propose a new search algorithm that allows traversing the proposed search space, and demonstrate that the space contains interesting architectures, which, when using identical training setup, outperform state-of-the-art hand-crafted architectures. Finally, a unified toolkit including a standardized training and evaluation protocol is introduced to foster reproducibility and enable fair comparison in NAS research. Overall, this work opens a pathway towards systematically exploring the full spectrum of neural architectures with a unified graph-based NAS perspective.

Figures (5)

• Figure 1: UniNAS search space spans a great variety of different network architectures, including ViT dosovitskiy2020image and EfficientNet tan2019efficientnet, which enables us to search for completely novel architectures. (500,000 networks from a random walk.)
• Figure 2: UniNAS architecture. We adopt a standard hierarchical design (e.g., dai2021coatnettu2022maxvit) with stem, four stages (each with several blocks) and a classification head but introduce a new type of basic building block UniNAS that integrates the universal directed acyclic graph; each of the UniNAS blocks has a different topology (see \ref{['sec:block']}). For clarity, we omit normalization and activation layers.
• Figure 3: UniNAS representation of an EfficientNet (tan2019rethinking) block with expansion ratio $4$ (left) and self-attention module (dosovitskiy2020image) with $2$ heads (right).
• Figure 4: FLOPs per elementary operation for a sample of 500,000 architectures generated via a random walk. A desirable property of the search step in \ref{['alg:search']} is that it should not collapse into a subspace of degraded networks. Our experiments confirm that the counts for each operation remain both stable and sufficiently variable, indicating that our random walk effectively explores large portions of the UniNAS space.
• Figure 5: UniNAS-A architecture. The architecture resembles two heads of a self-attention mechanism, but upon closer inspection, certain operations are added or removed.