Evolutionary Neural Architecture Search with Dual Contrastive Learning

TL;DR

The paper tackles the data-efficiency problem in predictor-guided evolutionary neural architecture search by introducing DCL-ENAS, a two-stage framework that leverages dual contrastive learning. A contrastive pretraining stage learns architecture representations from unlabeled data using a hard encoder to capture information flow and a soft encoder for predictive embeddings, while a contrastive fine-tuning stage optimizes a ranking-based predictor to guide evolution with limited labels. The method achieves state-of-the-art results on NASBench-101/201 benchmarks under constrained compute budgets and demonstrates practical improvements on an ECG time-series classification task, validating both efficiency and effectiveness. The work also introduces information-flow–driven evolutionary operators and a robust evaluating protocol, showing strong potential for resource-constrained NAS in real-world settings.

Abstract

Evolutionary Neural Architecture Search (ENAS) has gained attention for automatically designing neural network architectures. Recent studies use a neural predictor to guide the process, but the high computational costs of gathering training data -- since each label requires fully training an architecture -- make achieving a high-precision predictor with { limited compute budget (i.e., a capped number of fully trained architecture-label pairs)} crucial for ENAS success. This paper introduces ENAS with Dual Contrastive Learning (DCL-ENAS), a novel method that employs two stages of contrastive learning to train the neural predictor. In the first stage, contrastive self-supervised learning is used to learn meaningful representations from neural architectures without requiring labels. In the second stage, fine-tuning with contrastive learning is performed to accurately predict the relative performance of different architectures rather than their absolute performance, which is sufficient to guide the evolutionary search. Across NASBench-101 and NASBench-201, DCL-ENAS achieves the highest validation accuracy, surpassing the strongest published baselines by 0.05\% (ImageNet16-120) to 0.39\% (NASBench-101). On a real-world ECG arrhythmia classification task, DCL-ENAS improves performance by approximately 2.5 percentage points over a manually designed, non-NAS model obtained via random search, while requiring only 7.7 GPU-days.

Figures (10)

• Figure 1: Illustration of NAS and ENAS using image classification as an example.
• Figure 2: Overview of the proposed DCL-ENAS framework, which consists of a contrastive pretraining stage (CLP) and a contrastive fine-tuning with evolutionary search stage (CLF-ENAS).
• Figure 3: Illustration of the hard encoding process: each architecture is encoded into a unified binary vector based on a predefined path template and corresponding path identifiers (Path-ids).
• Figure 4: The architecture of the DCL-ENAS neural architecture predictor. Each candidate architecture is represented from two distinct perspectives, resulting in two fixed-length feature maps. These are subsequently processed through four layers—Concat, Project, Full, and Score—to produce a final ranking score. The final "Score" layer employs a Sigmoid activation function to normalize the output into the range $[0, 1]$. During the CLP stage, predictor parameters are updated up to the Concat layer, while the CLF stage introduces and optimizes new parameters specific to the Score layer.
• Figure 5: In search space, all paths will have a unique path index (Path-id). Therefore, for a neural structure, we can obtain the Path-id list $P_{\text{ids}}$ of this structure (the same as the hard-coded processing method, multi-level sorting of the path is required), and then the path index Path-id is embedded into the feature space as a Token. We stipulate that an architecture contains at most $L_{\text{seq}}$ paths. If the number of paths in an architecture is less than $L_{\text{seq}}$, a special token "$\langle$PAD$\rangle$" will be filled. The path identifier for the special token "$\langle$PAD$\rangle$" is a very large positive integer.

Theorems & Definitions (4)

• Proposition 3.1: Operation–code permutation invariance
• proof
• Definition 1
• Definition 2